A comprehensive guide into how we program machines to learn from images
· Follow
Published in · 10 min read · Oct 4, 2021
--
This will be a continuation from the introduction to AI, Machine Learning, and Deep Learning with Tensorflow story, where I discuss Standard Neural Networks. If this is all new for you, I recommend starting there first.
- Building a Digit Recognizer
- Evolving the Digit Recognizer
- What’s going on here?
- Building a Cats vs Dogs Classifier
MNIST Dataset
Computer Vision is the practice of using Neural Networks to learn image structure. When we look at an image, say of a shirt, how do we know this is a shirt? We can recognize it through identifying specific features of a shirt like the collar or the shape or context (where is the item worn?) shown in the image. This is similar to how we want to enable machines to learn images as well — we want models to understand the spatial context of images.
As before, let’s start with an example first. We can load the MNIST dataset from Tensorflow which is a large collection of handwritten digits.
The MNIST database contains 60,000 training images and 10,000 testing images, and the black and white images were normalized to fit into a 28x28 bounding box. This means when you load in the data it yields 60,000 28x28x1 items in a list, but the first layer in the model (Convolutional layer) will need a single 4D list that is 60,000x28x28x1 so this is why we reshape the training and testing images.
But why do we divide the images by 255.0? This is done to normalize the pixel values of images between 0 and 1, which makes it easier for the network to learn the optimal minimum.
Now to build the Convolutional Neural Network.
Our First Convolutional Neural Network
Standard Neural Networks just have Dense layers, so we now find 3 new layers with Convolutional Neural Networks: Conv2D, MaxPooling2D, and Flatten. These layers represent how this type of Neural Network can learn spatial features of an image.
We’ll cover what the layers are doing in a bit, but for now this model is able to scan the pixels of images through a moving filter, then flattens it into a single dimensional array which makes it similar to the Standard Neural Network, and finally sends through the Dense layers as we would with a non-convolutional network — a Dense layer with 128 nodes and an output layer with 10 nodes.
The final output layer contains a softmax activation which yields probabilities for the ten target classes (0–9 digit values) and it predicts the value with the highest probability.
Since this is a Classification task, specifically multi-classification, the loss function we’ll use is Sparse Categorical Cross Entropy. If it was binary classification, we’d use Binary Cross Entropy. This is in contrast to Regression tasks where we use Mean Squared Error.
We can use callbacks same as before to stop training when we hit an acceptable accuracy on the training set. Time to evaluate on unseen data.
Overall model performs pretty great, but slightly worse on the validation set so likely a little overfitting is happening to a minor degree. Dropout is a common technique to use so models don’t become too dependent on a given node, which helps with overfitting. In a nutshell, Dropout is a way to randomly drop some percentage of nodes in a given layer to force the model to not become too dependent on any given node.
When it comes to images, there is another technique as well: image augmentation. This involves, rotating, width/height shifting, shearing, zooming, cropping, and flipping. You can easily add this as a step before fitting the data to your model and it allows the model to generalize to new examples more effectively. The reason behind this is fairly straightforward too — if you had images of a dog but they were all with the dog standing on its legs, a model may perform worse with pictures where the dog is laying on its back. Image augmentation helps with this because it provides more unique orientations of images for it to learn from.
Let’s try to rebuild & evolve our initial model and see what we can do!
Loading & Preprocessing Set Up
The code looks the same from our initial model, except we’ve added a data generator for the training & validation sets to augment images by a certain degree (including normalizing the pixel values). A key thing to note here is that we don’t want to augment the testing data, just the training data. Why might that be? We want the model to generalize to new pictures that usually aren’t as skewed as our augmented images; we’re augmenting mainly to allow the model to account for variation in spatial positioning of pixels. This also creates a larger training set for us for our model to learn from.
Easily Scaling Up Complexity
Adding complexity with layers is as easy as it is with Standard Neural Networks, but key changes here include:
- Adding a Dropout layer that randomly drops 20% of the nodes from the hidden layer before the output
- Funneling the train/test images through a data generator which allows us to add easy preprocessing steps like image augmentation right when we go to train the model
- Setting batch size to 64 which is just the number of samples to process before the model gets updated per epoch. So 64 images get passed through at a time.
The model can be easily fit via the data generators as so:
I lowered the accuracy threshold to save on compute and time costs, but look at the training/testing loss and accuracy at each epoch — the model consistently is able to generalize to new data quite well.
A convolutional neural network is composed of filters that iterates over a section of an image to extract global and local spatial features of an image.
This represents 1 filter of size 3x3 being convolved on a 6x6 image. We often used 64 filters with a size of 5x5 to convolve over 28x28x1 images. Essentially, each cell gets multiplied by its counterpart and then summed to obtain the new value (3*1+1*1+2*1+0*0…=-5) and then the filter moves over 1 until all the values are scanned. With each of these layers, the model is attempting to learn discriminative global and local features of an image that resemble signals. Typically with more layers, the model can learn more specific features of an image. A good example is that the first layer of a model scanning an image of a face may learn the overall outline edges of a face, the second layer may learn the lines within the face that make out the eye from the nose and the mouth, the third layer may learn more distinctive features of eyes (shape, color, size, etc.), and so on.
An important thing to note here is that the values of the filter significantly dictate what gets detected in the output image. If we had all 0s in the filter, the output image would all be 0s as well, and no edge detection would be occurring. These filters are what the CNN learns as it’s able to extract meaningful, discriminative features in each image that allow it to minimize the loss in respect to the target labels as much as possible.
Beyond the Filter
You might notice how the output image shrinks after passing a filter through the original. This can happen at an escalating degree so if you want to have your output image to remain the same size, a common technique is to pad the original image with a layer of 0s around it.
Furthermore, maybe you don’t want to have the filter pass through an image stepping through one column over at a time. You can stride the convolution to skip two columns or more through an original image.
And then we have Pooling layers as well. These are typically Max or Average Pooling layers and they operate by taking the maximum or average of a section of an image.
Typically MaxPooling is done, but it’s a way to retain signals of features that are likely to be in a section of an image. Interestingly enough, MaxPooling doesn’t really have hyperparameters for Gradient Descent to learn from. It is a fixed value of filter size and stride that gets applied to each convolutional layer accordingly. Functionally, it’s a way of preserving feature information (signals indicated through high pixel value) in a region of an image with a smaller dimension of image.
To add even more complexity, we’re often working with colored images and not grayscale. This typically means images reside in 3 dimensions (RGB) so we’re not just passing a 2D convolution but a 3D one (5x5x3, for example) over a matrix of RGB pixel values.
Why Convolutions?
There are two main benefits to using Convolutional layers over Dense layers when operating over images:
- Parameter sharing: A feature detector that’s useful in one part of an image is likely useful in another part of an image (ex. vertical or horizontal edge detector)
- Sparsity of connections: In each layer, each output value depends only on a small number of inputs
So we have filter size, padding, strides, pooling filter size, number of layers, etc.(not even including the compute and time costs to train large scale models)— this is a lot to keep track of let alone edit to test what works well and what doesn’t! This is where a technique called Transfer Learning can be incredibly useful.
Transfer Learning is the concept of standing on the shoulders of giants; you are able to download popular and successful model architectures, along with their learned parameters, freeze all the layers except the output layer and run your data through it to obtain results. This has been proved to yield really successful results in a variety of domains, and often is the preferred choice when working with Deep Neural Network Architectures.
We can apply this knowledge to a cat vs. dog classifier.
Creating Data Directories
Tensorflow has a lot of handy functions that work well with structured directories. This initially requires us to load images into a folder path structure before building our model though. Technically this sub-section can be skipped if you only care about the modeling portion, but it’s a bit impractical to learn about modeling and not effective data design patterns.
We’ll load in Google’s Cats vs Dogs dataset before loading it into the appropriate directories.
This code reads in Google’s Cats and Dogs dataset into its appropriate directories. We can now build our preprocessing and dataflow generators.
Preprocessing Workflows
Previously we flowed the data via the train and test splits, this time we can simply set it up to flow from the directory for each. For context this only yields 2000 images in the training set and 1000 in the testing set. Now for the model!
We can easily load it in via the InceptionV3 function and qualify that we want the weights as well. Looping through the layers allows us to set their trainable flag to False.
Cherry Picking the Inception Network
This model is quite massive, I recommend checking out this site if you want to learn more of the Inception network: Inception-v3 Network Explained
We can take the output layer of the Inception network and then add on the last few standard layers of the Flatten(), Dense(), and Output layer with Dropout as we have done before. Since this is a binary classification (cat or dog), we only need one node in the output with a sigmoid activation function, and binary cross entropy for our loss function.
I lowered the epoch count significantly and the steps per epoch due to the miniscule dataset, and still the model performed incredibly well. In one epoch, we see ~96% accuracy.
That was a lot of ground covered today, but I attempted to provide a comprehensive starting guide to the fascinating world of computer vision here. A lot of the concepts build on the topics mentioned here, including creating AI art, enabling Autonomous Vehicles to “see”, extracting patterns from X-Ray or fMRI scans and more.
I’ll have a follow up piece that applies knowledge learned here to a dataset I’m interested in, but for now thanks for reading! Follow along for more on all things Data Science, Machine Learning, and AI.
References
[1] DeepLearning.AI, Convolutional Neural Networks
[2] DeepLearning.AI, Convolutional Neural Networks in Tensorflow
FAQs
Can you use TensorFlow for computer vision? ›
In this module, you will get an introduction to Computer Vision using TensorFlow. We'll use image classification to learn about convolutional neural networks, and then see how pre-trained networks and transfer learning can improve our models and solve real-world problems.
Is TensorFlow enough for machine learning? ›TensorFlow is an end-to-end open source platform for machine learning. TensorFlow is a rich system for managing all aspects of a machine learning system; however, this class focuses on using a particular TensorFlow API to develop and train machine learning models.
What is the difference between OpenCV and TensorFlow? ›The flexible architecture allows you to deploy computation to one or more CPUs or GPUs in a desktop, server, or mobile device with a single API. OpenCV belongs to "Image Processing and Management" category of the tech stack, while TensorFlow can be primarily classified under "Machine Learning Tools".
Is TensorFlow worth learning? ›TensorFlow Advantages:
With tons of flexibility, TensorFlow allows programmers to use this deep learning framework on any compatible device. As an open-source deep learning framework, TensorFlow is accessible and free to use, making it one of the most favorable framework options.
TensorFlow offers better visualization, which allows developers to debug better and track the training process. PyTorch, however, provides only limited visualization. TensorFlow also beats PyTorch in deploying trained models to production, thanks to the TensorFlow Serving framework.
Is TensorFlow good for image processing? ›TensorFlow is an open-source library for machine learning and deep learning applications. It has become a popular tool for image processing tasks due to its flexibility and scalability. In this guide, we will explore how to apply TensorFlow to image processing tasks.
What is the disadvantage of TensorFlow? ›1) Missing Symbolic loops: When we say about the variable-length sequence, the feature is more required.
Is TensorFlow still relevant? ›While PyTorch has become the de facto research framework after its explosive adoption by the research community and TensorFlow remains the legacy industry framework, there are certainly use cases for each in both domains.
Does anyone still use TensorFlow? ›It's shocking to see just how far TensorFlow has fallen. The 2022 state of competitive machine learning report came out recently and paints a very grim picture -- only 4% of winning projects are built with TensorFlow. This starkly contrasts with a few years ago, when TensorFlow owned the deep learning landscape.
Should I learn OpenCV before TensorFlow? ›OpenCV's documentation is much better (even if we're just talking about the DNN module here) than Tensorflow's C++ API's documentation. A final point where you might go with OpenCV instead of Tensorflow is that, with OpenCV, you can train an SVM model in C++.
Is TensorFlow better on CPU or GPU? ›
The Conclusion
While setting up the GPU is slightly more complex, the performance gain is well worth it. In this specific case, the 2080 rtx GPU CNN trainig was more than 6x faster than using the Ryzen 2700x CPU only. In other words, using the GPU reduced the required training time by 85%.
The OpenCV software has become a de-facto standard tool for all things related to Computer Vision. In 2023, OpenCV is still highly popular, with over 29'000 downloads every week. OpenCV is written in C and C++.
Is Google dropping TensorFlow? ›With companies and researchers leaving Tensorflow and going to PyTorch, Google seems to be interested in moving its products to JAX, addressing some pain points from Tensorflow like the complexity of API, and complexity to train in custom chips like TPU.
How many days will it take to learn TensorFlow? ›How Long Does it Take to Learn TensorFlow? If you already know Python programming and the theoretical foundations of neural networks, you can become a productive TensorFlow developer in 1 to 2 months. If you are a complete beginner in machine learning and programming, 3-6 months is a more realistic timeline.
Why is TensorFlow difficult? ›TensorFlow is considered both difficult to learn and use, largely due to the amount of programming skill needed. While TensorFlow is powerful and streamlines the development and training of machine learning models, the power that TensorFlow delivers requires extensive knowledge of how to use it.
Does Tesla use TensorFlow or PyTorch? ›Tesla uses PyTorch for Autopilot, their self-driving technology. The company uses PyTorch to train networks to complete tasks for their computer vision applications, including object detection and depth modeling.
Which is the best platform to learn computer vision? ›- MathWorks. ...
- DeepLearning.AI. DeepLearning.AI TensorFlow Developer. ...
- Free. Edge Impulse. ...
- Coursera Project Network. Deep Learning with PyTorch : Image Segmentation. ...
- DeepLearning.AI, Stanford University. Machine Learning. ...
- University at Buffalo. Computer Vision Basics. ...
- University of Toronto. Self-Driving Cars. ...
- IBM. IBM Machine Learning.
- OpenCV. A software library for machine learning and computer vision is called OpenCV. ...
- Viso Suite. ...
- CUDA. ...
- MATLAB. ...
- Keras. ...
- SimpleCV. ...
- BoofCV. ...
- CAFFE.
Very useful for reliable face recognition when there is a very large number of faces to identify. Triplet loss: A loss function that's used when the only type of training data available are pair similarities.
Does Nvidia use TensorFlow? ›TensorFlow is written both in optimized C++ and the NVIDIA® CUDA® Toolkit, enabling models to run on GPU at training and inference time for massive speedups.
What GPU is needed for TensorFlow? ›
Hardware requirements
NVIDIA® GPU card with CUDA® architectures 3.5, 5.0, 6.0, 7.0, 7.5, 8.0 and higher.
What pretty much everyone already knew was gonna happen, is now happening -- JAX is being gradually rolled out to replace TensorFlow (at least for internal use at Google). After losing out to PyTorch, Google is quietly moving to roll out a new AI framework internally called JAX.
Do companies use TensorFlow? ›Who uses TensorFlow? 517 companies reportedly use TensorFlow in their tech stacks, including Uber, Delivery Hero, and Hepsiburada.
Does TensorFlow have future? ›Despite Google's significant investment in TensorFlow, the mistake of its design is so fundamental that it cannot be redone. Huang predicts that in the future, few people will still use TensorFlow, making this mistake one of the most costly Google has ever made in AI.
Should I learn PyTorch or TensorFlow 2023? ›TensorFlow is good at deploying models in production to build AI products, while PyTorch is preferred in academia for research tasks. Thus, both TensorFlow and PyTorch are good frameworks to learn.
Which big companies use TensorFlow? ›Companies using Google TensorFlow for Machine Learning and Data Science Platform include: Anthem, Inc., a United States based Healthcare organisation with 83400 employees and revenues of $121.87 billion, Intel Corporation, a United States based Manufacturing organisation with 121100 employees and revenues of $63.10 ...
Does Disney use PyTorch? ›We use a custom PyTorch IterableDataset that, in combination with PyTorch's DataLoader, allows us to read different parts of the video with parallel CPU workers. The video is split in chunks based on its I-frames and each worker reads different chunks.
Is TensorFlow backed by Google? ›TensorFlow was developed by the Google Brain team for internal Google use in research and production. The initial version was released under the Apache License 2.0 in 2015.
Why did Google release TensorFlow? ›The most widely spreaded theory is that Google open-sourced TensorFlow to connect users to their cloud services. It makes perfect sense. TensorFlow is there to make a technical impact and attract users, and the Google Cloud Platform (GCP) would be the best premium service for using TensorFlow.
Is TensorFlow owned by Google? ›TensorFlow is an open source framework developed by Google researchers to run machine learning, deep learning and other statistical and predictive analytics workloads.
Is TensorFlow just for deep learning? ›
TensorFlow is an open-source library developed by Google primarily for deep learning applications. It also supports traditional machine learning. TensorFlow was originally developed for large numerical computations without keeping deep learning in mind.
Is PyTorch faster than TensorFlow? ›In general, TensorFlow and PyTorch implementations show equal accuracy. However, the training time of TensorFlow is substantially higher, but the memory usage was lower. PyTorch allows quicker prototyping than TensorFlow, but TensorFlow may be a better option if custom features are needed in the neural network.
Do you need to know math for TensorFlow? ›It is important to understand mathematical concepts needed for TensorFlow before creating the basic application in TensorFlow. Mathematics is considered as the heart of any machine learning algorithm. It is with the help of core concepts of Mathematics, a solution for specific machine learning algorithm is defined.
How much faster is TensorFlow with GPU? ›GPU-Accelerated TensorFlow
TensorFlow runs up to 50% faster on the latest Pascal GPUs and scales well across GPUs. Now you can train the models in hours instead of days.
CPUs are less efficient than GPUs for deep learning because they process tasks in order one at a time. As more data points are used for input and forecasting, it becomes more difficult for a CPU to manage all of the associated tasks.
Which Python version is best for TensorFlow GPU? ›Python version 3.4+ is considered the best to start with TensorFlow installation. Consider the following steps to install TensorFlow in Windows operating system.
What is the disadvantage of OpenCV? ›The movement of head or different camera positions can cause changes of facial texture and it will generate the wrong result. Occlusion means the face as beard, mustache, accessories (goggles, caps, mask, etc.)
Is OpenCV better in Python or C++? ›Python is significantly slower than C++ with opencv, even for trivial programs. The most simple example I could think of was to display the output of a webcam on-screen and display the number of frames per second. With python, I achieved 50FPS (on an Intel atom). With C++, I got 65FPS, an increase of 25%.
Is TensorFlow dead 2023? ›TensorFlow is Not Dead, Yet
Google learned from Meta's PyTorch and made TensorFlow 2.0, which is better and easier for research than its previous version. Still, researchers have no reason to return to giving TensorFlow another chance.
Amazon SageMaker enables developers and data scientists to quickly and easily build, train, and deploy machine learning models at any scale. Amazon SageMaker removes all the barriers that typically slow down developers who want to use machine learning.
Does DeepMind use TensorFlow or PyTorch? ›
The Google DeepMind AI project started out using Torch, and then switched to TensorFlow.
Is TensorFlow exam hard? ›It's impossible to pass the exam without true knowledge of TensorFlow and Deep Learning! In order to get Certified in TensorFlow, you have to: Pass a 5-hour test administered by Google and TensorFlow. Be deeply familiar (no pun intended) with all aspects of Deep Learning and advanced machine learning concepts.
Is scikit learn easier than TensorFlow? ›In practice, Scikit-learn is utilized with a wide range of models. It provides under-the-hood specialization optimization, making it easier to compare neural network models and TensorFlow models. It is possible to compare completely distinct variants of machine learning models using Scikit-learn.
What should I learn before TensorFlow? ›- Deep learning platforms.
- Data flow graph.
- Linear regression using TensorFlow.
- Recurrent neural networks.
- TensorFlow Object Detection API.
TensorFlow has become a preferred deep learning library at Uber for a variety of reasons. To start, the framework is one of the most widely used open source frameworks for deep learning, which makes it easy to onboard new users.
Which language is best for TensorFlow? ›Python is the recommended language for TensorFlow, although it also uses C++ and JavaScript. Python was developed to help programmers write clear, logical code for both small and large projects. It's often used to build websites and software, automate tasks, and carry out data analysis.
Is TensorFlow beginner friendly? ›TensorFlow makes it easy for beginners and experts to create machine learning models for desktop, mobile, web, and cloud. See the sections below to get started.
Which Python framework is best for computer vision? ›- OpenCV. With over 2500 optimized image and video processing algorithms, OpenCV is one of the most widely used computer vision libraries for deploying computer vision applications. ...
- TensorFlow. ...
- SimpleCV. ...
- Caffe. ...
- PyTorch. ...
- Keras. ...
- Detectorn2.
Computer vision algorithms analyze certain criteria in images and videos, and then apply interpretations to predictive or decision making tasks. Today, deep learning techniques are most commonly used for computer vision.
Which tool is used for computer vision? ›TensorFlow is one of the most well-known end-to-end open-source machine learning platforms, which offers a vast array of tools, resources, and frameworks. TensorFlow is beneficial for developing and implementing machine learning-based computer vision applications.
Is computer vision easier than NLP? ›
NLP is language-specific, but CV is not.
However, computer vision is much easier. Take pedestrian detection, for example. An image taken in the US and in China usually has no significant difference. An ML model trained with images taken in China usually works also well on images taken in the US.
Computer vision offers the ability to sense surroundings and process the information it's taken in. Likewise, NLP enables the understanding of spoken or written language—and knowing which words to string together to communicate a prescribed message, much the same way as humans do.
Should I use Python or C++ for computer vision? ›Programming Languages Best Suited for Computer Vision
However, Python's runtime is slowed because libraries like OpenCV are created in C++. After all, it still makes calls to C/C++ libraries. This implies Python will provide a development edge while C++ will provide performance optimization.
Which is the top GPU for deep learning? NVIDIA, the market leader, offers the best deep-learning GPUs in 2022. The top NVIDIA models are Titan RTX, RTX 3090, Quadro RTX 8000, and RTX A6000.
What programming language should I learn for computer vision? ›We have several programming language choices for computer vision – OpenCV using C++, OpenCV using Python, or MATLAB. However, most engineers have a personal favourite, depending on the task they perform. Beginners often pick OpenCV with Python for its flexibility.
Which is better computer vision or deep learning? ›The advancement of technologies pertaining to deep learning has made it possible to construct computer vision models that are both more accurate and complicated. The incorporation of computer vision applications is becoming increasingly beneficial as these technologies continue to advance.
Is OpenCV the best for computer vision? ›1. OpenCV. OpenCV is the oldest and by far the most popular open-source computer vision library, which aims at real-time vision. It's a cross-platform library supporting Windows, Linux, Android, and macOS and can be used in different languages, such as Python, Java, C++, etc.
What are the three types of computer vision? ›Different types of computer vision include image segmentation, object detection, facial recognition, edge detection, pattern detection, image classification, and feature matching.
Is computer vision used in Google Maps? ›Google Maps immersive view feature uses AI and computer vision to fuse Street View and aerial images. This creates a rich, digital model of places and the added layers of information such as weather, traffic, and how busy a place is essentially makes it a holistic offering.
Is computer vision high paying? ›Highest salary that a Computer Vision Engineer can earn is ₹21.7 Lakhs per year (₹1.8L per month). How does Computer Vision Engineer Salary in India change with experience? An Entry Level Computer Vision Engineer with less than three years of experience earns an average salary of ₹7.6 Lakhs per year.
Is computer vision hard? ›
But it's still a really hard problem that requires knowledge, not just data. The human brain can connect the dots based on information adjacency - how contextually close pieces of information are to each other - but this is learnt over time and can be hard to teach a computer.
Is computer vision still relevant? ›Computer vision is used in industries ranging from energy and utilities to manufacturing and automotive – and the market is continuing to grow. It is expected to reach USD 48.6 billion by 2022.