{"id":416,"date":"2023-07-11T03:58:03","date_gmt":"2023-07-11T03:58:03","guid":{"rendered":"https:\/\/themenectar.com\/salient\/mag\/?p=416"},"modified":"2026-03-27T09:16:37","modified_gmt":"2026-03-27T09:16:37","slug":"convolutions-to-cnns-deep-learning","status":"publish","type":"post","link":"https:\/\/curriculo.me\/engineering\/convolutions-to-cnns-deep-learning\/","title":{"rendered":"Convolutions to CNNs: From Array Operations to Deep Learning"},"content":{"rendered":"[vc_row type=”in_container” full_screen_row_position=”middle” column_margin=”default” column_direction=”default” column_direction_tablet=”default” column_direction_phone=”default” scene_position=”center” text_color=”dark” text_align=”left” row_border_radius=”none” row_border_radius_applies=”bg” row_position_desktop=”default” row_position_tablet=”inherit” row_position_phone=”inherit” overflow=”visible” overlay_strength=”0.3″ gradient_direction=”left_to_right” shape_divider_position=”bottom” bg_image_animation=”none”][vc_column column_padding=”no-extra-padding” column_padding_tablet=”inherit” column_padding_phone=”inherit” column_padding_position=”all” flex_gap_desktop=”10px” column_element_direction_desktop=”default” column_element_spacing=”default” desktop_text_alignment=”default” tablet_text_alignment=”default” phone_text_alignment=”default” background_color_opacity=”1″ background_hover_color_opacity=”1″ column_backdrop_filter=”none” column_shadow=”none” column_border_radius=”none” column_link_target=”_self” column_position=”default” gradient_direction=”left_to_right” overlay_strength=”0.3″ width=”1\/1″ tablet_width_inherit=”default” animation_type=”default” bg_image_animation=”none” border_type=”simple” column_border_width=”none” column_border_style=”solid”][vc_column_text]In these application areas, convolution is often performed as a filter that transforms signals and pixels into more desirable values.<\/p>\n

In high-performance computing, the convolution pattern is often referred to as stencil computation. Convolution typically involves a significant number of arithmetic operations on each data element. Each output data element can be calculated independently of each other, a desirable trait for parallel computing. On the other hand, there is substantial level of input data sharing among output data elements with somewhat challenging boundary conditions. This makes convolution an important use case of sophisticated tiling methods and input data staging methods.<\/p><\/blockquote>\n

Convolution is an array operation where each output data element is a weighted sum of a collection of neighboring input elements. The weights used in the weighted sum calculation are defined by an input mask array, commonly referred to as the convolution kernel. We will refer to these mask arrays as convolution masks. The same convolution mask is typically used for all elements of the array<\/p>\n

Below is shown a convolution example for 1D data where a 5-element convolution mask array M is applied to a 7-element input array N.<\/p>\n

\"1D<\/figure>\n

For image processing and computer vision, input data are typically two-dimensional arrays, with pixels in an x-y space. Image convolutions are therefore 2D convolutions. The mask does not have to be a square array.<\/p>\n

\"2D<\/figure>\n
\n

1. From 1D to 2D convolution<\/h2>\n

You know 1D convolution:<\/p>\n

    \n
  • Input:<\/strong> 1D array (e.g., [x0, x1, x2, ...]<\/code>)<\/li>\n
  • Kernel:<\/strong> small 1D array (e.g., [w0, w1, w2]<\/code>)<\/li>\n
  • Operation:<\/strong> slide the kernel and take weighted sums.<\/li>\n<\/ul>\n

    For images<\/strong>, the input is 2D (height \u00d7 width), so:<\/p>\n

      \n
    • Input:<\/strong> matrix of pixels, shape H \u00d7 W<\/code><\/li>\n
    • Kernel:<\/strong> small matrix, e.g. 3 \u00d7 3<\/code> or 5 \u00d7 5<\/code><\/li>\n
    • Operation:<\/strong> place the kernel over each 3 \u00d7 3<\/code> patch of the image, multiply element wise, sum \u2192 one output pixel.<\/li>\n<\/ul>\n

      This is just the same idea you already know, but:<\/p>\n

        \n
      • Array \u2192 2D array (matrix).<\/li>\n
      • Neighboring \u201celements\u201d \u2192 neighboring pixels in 2D.<\/li>\n<\/ul>\n
        \n

        2. Multiple channels (e.g., RGB)<\/h2>\n

        Real images are not just 2D, they often have channels<\/strong>:<\/p>\n

          \n
        • Shape: H \u00d7 W \u00d7 C<\/code> (e.g., C = 3<\/code> for R, G, B).<\/li>\n
        • Then a kernel<\/strong> also has depth C<\/code>:\n
            \n
          • Shape: kH \u00d7 kW \u00d7 C<\/code>.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n

            Convolution step:<\/p>\n

              \n
            • For one position, you take a small cube: kH \u00d7 kW \u00d7 C<\/code>.<\/li>\n
            • Multiply elementwise with the kernel kH \u00d7 kW \u00d7 C<\/code>.<\/li>\n
            • Sum all values \u2192 1 number (one output channel value at that location).<\/li>\n<\/ul>\n

              So conceptually: 3D convolution over space + channels<\/strong>, but still just \u201cweighted sum of neighbors\u201d.<\/p>\n

              \n

              3. Filters (kernels) and feature maps<\/h2>\n

              In CNNs, you don\u2019t use just one kernel; you use many<\/strong>:<\/p>\n

                \n
              • Suppose you use F<\/code> different kernels.<\/li>\n
              • Each kernel has shape kH \u00d7 kW \u00d7 C_in<\/code>.<\/li>\n
              • Each kernel produces one output channel<\/strong>, called a feature map<\/strong>.<\/li>\n<\/ul>\n

                So if the input shape is:<\/p>\n

                  \n
                • H \u00d7 W \u00d7 C_in<\/code>,<\/li>\n<\/ul>\n

                  and you use F<\/code> kernels, the output shape becomes:<\/p>\n

                    \n
                  • H_out \u00d7 W_out \u00d7 F<\/code>.<\/li>\n<\/ul>\n

                    Each of those F<\/code> output channels captures a different kind of pattern:<\/p>\n

                      \n
                    • One kernel may learn to detect edges<\/strong>, another corners<\/strong>, another textures<\/strong>, etc.<\/li>\n<\/ul>\n

                      Think:
                      \n\u201cA CNN layer = apply many different convolutions (filters) over the input.\u201d<\/strong><\/p>\n

                      \n

                      4. Stride and padding<\/h2>\n

                      Two more array-related details:<\/p>\n

                        \n
                      • Stride:<\/strong> how far you move the kernel each step.\n
                          \n
                        • Stride 1: move 1 pixel at a time \u2192 dense output.<\/li>\n
                        • Stride 2: skip every other pixel \u2192 smaller output.<\/li>\n<\/ul>\n<\/li>\n
                        • Padding:<\/strong> add zeros around the border of the image.\n
                            \n
                          • Without padding, output gets smaller.<\/li>\n
                          • With padding, you can keep the same height\/width.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n

                            Mathematically, it\u2019s still just sliding windows and dot products, but with control over:<\/p>\n

                              \n
                            • How far you slide (stride).<\/li>\n
                            • Whether you lose border information (padding).<\/li>\n<\/ul>\n
                              \n

                              5. Nonlinearity: adding activation functions<\/h2>\n

                              So far it\u2019s all linear<\/strong>: weighted sums. A CNN layer adds nonlinearity<\/strong>:<\/p>\n

                                \n
                              • After convolution, apply an activation function elementwise
                                \n(e.g., ReLU(x) = max(0, x)<\/code>).<\/li>\n<\/ul>\n

                                So a single conv layer<\/strong> is:<\/p>\n

                                  \n
                                1. Convolution (linear weighted sums).<\/li>\n
                                2. Activation (nonlinear).<\/li>\n<\/ol>\n

                                  This combination lets CNNs learn complex, non-linear mappings.<\/p>\n

                                  \n

                                  6. Stacking layers \u2192 deep feature hierarchies<\/h2>\n

                                  A full CNN is just many layers<\/strong> of:<\/p>\n

                                    \n
                                  • Convolution \u2192 activation (\u2192 sometimes pooling).<\/li>\n<\/ul>\n

                                    Intuition:<\/p>\n

                                      \n
                                    • Early layers<\/strong>: detect simple things (edges, blobs).<\/li>\n
                                    • Middle layers<\/strong>: combine them into textures, shapes.<\/li>\n
                                    • Later layers<\/strong>: combine shapes into object parts and objects.<\/li>\n<\/ul>\n

                                      All of this is still built from the same primitive you understand:
                                      \n\u201ctake local neighborhoods and compute weighted sums.\u201d<\/strong><\/p>\n

                                      \n

                                      7. Connection to fully connected (dense) layers<\/h2>\n

                                      A fully connected layer<\/strong> is also just weighted sums:<\/p>\n

                                        \n
                                      • Input: 1D vector.<\/li>\n
                                      • Output unit = sum of (weight \u00d7 input) + bias.<\/li>\n<\/ul>\n

                                        Difference:<\/p>\n

                                          \n
                                        • Dense layer: every output unit uses all<\/strong> input elements.<\/li>\n
                                        • Conv layer: each output unit uses only a local neighborhood<\/strong> (sparse connectivity) and shares weights<\/strong> across positions.<\/li>\n<\/ul>\n

                                          So you can see CNNs as:<\/p>\n

                                            \n
                                          • A clever way to use your convolution operation to get:\n
                                              \n
                                            • Locality (nearby pixels matter more).<\/li>\n
                                            • Weight sharing (same kernel across all positions).<\/li>\n
                                            • Fewer parameters than a dense layer on full images.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
                                              \n

                                              Conclusion: From Simple Operation to Powerful Networks<\/h2>\n
                                              \"At each position, take neighbors and compute a weighted sum with a kernel,\"<\/code><\/pre>\n
                                              then you\u2019re already standing on the core idea behind CNNs.<\/p>\n
                                              CNNs take that simple building block and:<\/p>\n
                                                \n
                                              • Extend it from 1D arrays to 2D\/3D tensors.<\/li>\n
                                              • Apply many different kernels in parallel to produce rich feature maps.<\/li>\n
                                              • Add nonlinearities to escape pure linearity.<\/li>\n
                                              • Stack many such layers so that each stage \u201csees\u201d more abstract structure.<\/li>\n<\/ul>\n
                                                The magic of CNNs isn\u2019t a new kind of math\u2014it\u2019s the composition of many small, local, convolution operations into a deep architecture that can learn powerful visual representations.[\/vc_column_text][\/vc_column][\/vc_row]\n","protected":false},"excerpt":{"rendered":"
                                                We will study convolution, which is a popular array operation that is used in various forms in signal processing, digital recording, image processing, video processing, and computer vision.<\/p>\n","protected":false},"author":3,"featured_media":862,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[12,9],"tags":[22,20,18,21],"class_list":{"0":"post-416","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-software","8":"category-tech","9":"tag-automation","10":"tag-cnn","11":"tag-data-science","12":"tag-deep-learning"},"_links":{"self":[{"href":"https:\/\/curriculo.me\/engineering\/wp-json\/wp\/v2\/posts\/416","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/curriculo.me\/engineering\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/curriculo.me\/engineering\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/curriculo.me\/engineering\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/curriculo.me\/engineering\/wp-json\/wp\/v2\/comments?post=416"}],"version-history":[{"count":6,"href":"https:\/\/curriculo.me\/engineering\/wp-json\/wp\/v2\/posts\/416\/revisions"}],"predecessor-version":[{"id":872,"href":"https:\/\/curriculo.me\/engineering\/wp-json\/wp\/v2\/posts\/416\/revisions\/872"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/curriculo.me\/engineering\/wp-json\/wp\/v2\/media\/862"}],"wp:attachment":[{"href":"https:\/\/curriculo.me\/engineering\/wp-json\/wp\/v2\/media?parent=416"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/curriculo.me\/engineering\/wp-json\/wp\/v2\/categories?post=416"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/curriculo.me\/engineering\/wp-json\/wp\/v2\/tags?post=416"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}