{"id":131,"date":"2023-07-08T07:36:00","date_gmt":"2023-07-08T07:36:00","guid":{"rendered":"https:\/\/themenectar.com\/salient\/mag\/?p=131"},"modified":"2026-03-27T09:06:10","modified_gmt":"2026-03-27T09:06:10","slug":"lessons-learned-from-professional-challenges","status":"publish","type":"post","link":"https:\/\/curriculo.me\/engineering\/lessons-learned-from-professional-challenges\/","title":{"rendered":"The Architecture of Information: How I Learned to Make Messy Data Readable"},"content":{"rendered":"\n

Not long ago, my system hit a breaking point.<\/p>\n\n\n\n

It wasn\u2019t a crash; it was a trust problem<\/strong>. I watched a client dataset get ingested, and instead of a clean structure, the system created a mess of duplicates. The same organization appeared three times in the knowledge graph because the source data used three different naming conventions: \u201cMcKinsey & Company,\u201d<\/em> \u201cMcKinsey and Company,\u201d<\/em> and \u201cMckinsey.\u201d<\/em> To a human, it\u2019s obviously one entity. To a graph, it was three separate nodes with three disconnected sets of relationships. Three parallel versions of the same truth.<\/p>\n\n\n\n

That was the moment I stopped thinking about data as something you simply store and started thinking about it as something you architect<\/strong>.<\/p>\n\n\n\n

Everything Is Messier Than the Schema Suggests<\/strong><\/h3>\n\n\n\n

Here\u2019s what nobody prepares you for when you build systems that process human-generated data: humans are wildly inconsistent. And I don\u2019t mean that in a charming way; I mean it in a way that breaks every assumption your schema makes.<\/p>\n\n\n\n

One person writes “Python.” Another writes “Python Programming Language.” A third writes “python” in lowercase. To a naive string comparison, these are four completely different entities. Four nodes. Four lies in your graph.<\/p>\n\n\n\n

I didn\u2019t set out to solve this problem; I just wanted to build a clean data pipeline. But when your pipeline ingests human data, this mess is<\/em> the problem. Everything else is downstream.<\/p>\n\n\n\n

Visual Hierarchy as an Engineering Protocol<\/strong><\/h3>\n\n\n\n

Before I go deep on the technical architecture, let me make a case for something engineers don\u2019t talk about enough: how information looks matters as much as what it contains.<\/strong><\/p>\n\n\n\n

When you glance at a well-structured report or dashboard, your brain builds a mental model in under three seconds. You instinctively know where the key data lives and the relative importance of each section. This isn’t “design”\u2014it’s a data communication protocol.<\/strong><\/p>\n\n\n\n

The way you present data shapes the decisions people make from it. When I\u2019m building a system to structure information, I can\u2019t just think about schema correctness. I have to think about perceptual weight.<\/strong> What does a human\u2019s eye land on first? How do I encode those priorities into a data structure that a rendering engine can actually act on?<\/p>\n\n\n\n

Knowledge Graphs: Relationships Are the Data<\/strong><\/h3>\n\n\n\n

A database table stores facts. A knowledge graph stores meaning<\/strong>.<\/p>\n\n\n\n

In my graph, every piece of information is a node, but the actual intelligence lives in the edges<\/strong>:<\/p>\n\n\n\n

    \n
  • HAS_ATTRIBUTE: Links an entity to its specific, contextual properties.<\/li>\n\n\n\n
  • BELONGS_TO: Associates a record with its parent organization, which has its own properties and relationships.<\/li>\n\n\n\n
  • REQUIRES: Connects a project or record to the specific technologies or skills involved.<\/li>\n<\/ul>\n\n\n\n

    When data is linked this way, it isn’t just a list; it\u2019s a narrative<\/strong>. It emerges naturally from the structure\u2014I don\u2019t have to write special “narrative generation” code. The relationships are<\/em> the story.<\/p>\n\n\n\n

    Data Density: The Constraint is the Feature<\/strong><\/h3>\n\n\n\n

    I\u2019m obsessed with Data Density<\/strong>\u2014meaningful information per unit of visual space. A dense document isn\u2019t a cluttered one; it\u2019s one where every element earns its position. In my system, density is enforced through strict structural constraints:<\/p>\n\n\n\n

      \n
    • items_per_category: 5: You don’t list 47 attributes. You list the five that matter. The constraint forces prioritization.<\/li>\n\n\n\n
    • summary_max_sentences: 2: Two sentences. That\u2019s it. Say what it is and why it matters. Everything else is noise.<\/li>\n\n\n\n
    • forbidden_filler_phrases: I wrote actual validation logic that rejects empty words like “best-in-class”<\/em> or “synergy.”<\/em> They take up space and communicate nothing.<\/li>\n<\/ul>\n\n\n\n

      The Standardization Stack: The Plumbing Nobody Sees<\/strong><\/h3>\n\n\n\n

      To solve that “McKinsey” problem, I built a four-layer stack to ensure data integrity:<\/p>\n\n\n\n

        \n
      1. Normalization:<\/strong> Cleaning names on ingestion and stripping verbose suffixes.<\/li>\n\n\n\n
      2. Fuzzy Deduplication:<\/strong> Using a 0.82 similarity threshold\u2014high enough to catch “Docker Container” and “Docker,” but low enough to keep “Go” and “Git” separate.<\/li>\n\n\n\n
      3. Deterministic IDs:<\/strong> Every node gets an ID derived from its content (e.g., attr_python). Process the same messy input twice, get the same clean graph once.<\/li>\n\n\n\n
      4. The Mutation Engine:<\/strong> A dispatch system that handles updates and corrections without losing the provenance of where the data came from.<\/li>\n<\/ol>\n\n\n\n

        What I Learned from Failure<\/strong><\/h3>\n\n\n\n

        This system has failed in ways that taught me more than the successes. It once merged “Machine Learning” and “Machine Operation” because they shared a high similarity score. That failure taught me the massive gap between “structurally valid” and “semantically meaningful.”<\/p>\n\n\n\n

        I also learned to treat automated extraction as untrusted input.<\/strong> I now use noise-pattern filters to ensure the system doesn’t “hallucinate” records that sound plausible but don’t exist.<\/p>\n\n\n\n

        The Bottom Line<\/strong><\/h3>\n\n\n\n

        The architecture of information is the architecture of trust.<\/strong> When data is normalized and semantically linked, people trust the output. When it\u2019s full of orphans and duplicates, the substance doesn’t matter.<\/p>\n\n\n\n

        The job isn’t just to write code; it’s to turn chaos into something queryable. If you want to turn noise into knowledge, you don’t just need storage. You need architecture<\/p>\n\n\n\n

        <\/figure>\n\n\n\n