This is the last piece of the 5-part series on Range Is The New Black. Previously published posts are linked at the bottom. Ok, Onward.

This closing chapter is also perhaps the most liberating one of all.

If you’ve followed the arc of this series so far, you will see that the interpretation of the final chapter of Range as it applies to career development doesn’t just reinforce the previous ideas. It reframes the very goal of a career. Not as a linear climb from novice to expert, but as an evolving, curious journey led by a mindset that evolution is essential for growth.

An evolutionary mindset perhaps never more important than it is now, as human brilliance is squarely staring at machine brilliance and what it is becoming capable of doing.

Chapter 12: Deliberate Amateurs

David introduces us to the concept of the deliberate amateur. Someone who, even after achieving deep expertise, deliberately chooses to approach problems as if they’re new again. Not because they don’t know better, but because they do.

They know that fresh thinking comes not from mastery alone, but from resisting the rigidity that mastery can create.

To illustrate this, David brings us the story of Frances Hesselbein, who took over the Girl Scouts of the USA in the 1970s. She had no formal management training. No MBA. No pedigree in organizational transformation.

Yet she modernized the Girl Scouts into one of the most respected leadership development organizations in the country.

Her secret? She didn’t try to be the smartest person in the room. She asked good questions. She ignored “best practices.” She trusted her outsider status to rethink what leadership could look like.

In fact, her amateurism wasn’t a weakness. It was her superpower.

What This Means for Data & Tech Professionals in 2026

In a world where AI can learn the rules of any domain quickly, the last thing you want to become is a rule-bound expert who forgets how to ask “why.”

Here’s the paradox: the deeper you go into your technical craft, the more likely you are to become rigid in your mental models. And the more dangerous that rigidity is likely to become in an environment shaped by exponential change.

That’s why some of the best leaders in tech deliberately cultivate an amateur mindset, even when they’re operating at the top of their game.

They are, for the purpose of illustration:

• The Head of Engineering who regularly joins onboarding sessions for junior engineers to hear how beginners think.

• The Senior Staff Data Scientist who picks up visual design to better communicate model insights.

• The Principal Architect who shadows product managers, not to advise but to listen and relearn user needs.

These professionals aren’t becoming less expert. They’re choosing range over rigidity. They’re choosing curiosity over correctness.

The Power of Not Knowing (On Purpose)

In the AI era, where much of the “doing” is being automated, the real leverage comes from thinking differently i.e., combining, questioning, reframing, reimagining.

That doesn’t happen when you’re always the expert in the room. It happens when you choose to occasionally become the beginner again.

David shows how deliberate amateurs across fields from Nobel-winning scientists to elite artists stay relevant by refusing to box themselves into a single identity. They wander. They tinker. They play.

And that’s not a detour. That’s the fodder for innovation.

Practical Strategy: Become a Career Amateur (Deliberately)

1. Carve out “amateur hours”

   Block time each week for exploration outside your core domain. Study fields adjacent to your own. Pick up an unrelated skill, not for career gain, but for mental flexibility.

2. Join teams where you’re not the expert

   Deliberately embed yourself in projects where others know more. Watch how you learn. Observe your own biases. That’s where you’ll grow.

3. Ask “why” like a beginner

   In meetings, practice asking “why are we doing it this way?” Not to challenge, but to uncover assumptions. Often, you’ll surface outdated thinking others missed.

4. Write and teach across disciplines

   Explaining a technical concept to non-technical peers forces you to see it with new eyes. It’s a powerful way to stay mentally limber.

5. Normalize saying “I don’t know”

   Especially in leadership roles, this is powerful. It gives your team permission to explore, learn, and challenge assumptions, which is exactly what you want in an AI-enabled environment.

Bottom Line: Expertise Is a Starting Point. Not a Destination.

The deliberate amateur doesn’t reject mastery. They build on it, but refuse to let it become a cage. They use their range across roles, industries, and ways of thinking to stay creative, connected, and adaptable.

And in the modern tech world, that’s what leadership looks like.

The ability to learn publicly. To think laterally. To ask “what if?” long after others have stopped.

Conclusion: Why Range Is the New Black

We opened this series with a bold claim: that in a world reshaped by AI, automation, and complexity, range, not narrow specialization is becoming the most important advantage a professional can build.

And now, having walked through the book Range, it’s clear: range isn’t just a philosophy. It’s a playbook.

Across every chapter, one truth keeps repeating: the most resilient, creative, and influential professionals are those who refuse to be defined by a single skillset or a static path.

They sample broadly.

They adapt relentlessly.

They question the rules.

They think like outsiders.

They update like foxes.

They drop tools when needed.

They lead like deliberate amateurs.

Whether you’re a senior engineer wondering how to stay relevant, a data professional hoping to step into strategic influence, or a team leader stepping into leading an area of large-scale transformation programs, range is no longer optional. It’s the new foundation of career durability and exponential growth.

So don’t chase the narrowest lane. Build your bridge wide. Cultivate your curiosity. Own your unexpected combinations. The future belongs to those who can flex.

Range is the new black. And, this series is your career development playbook.

Thanks for being part of this journey. If you found this series valuable, share it with someone who will appreciate this perspective and needs to hear the most. The next chapter of your own range might be closer than you think.

To your success,

Vijay

PS. If you missed the previous posts, start from here.

This next topic brings us to one of the most disruptive and empowering ideas in Range. The idea that sometimes, not being “qualified” is your biggest advantage. Especially when you’re solving a problem no one else has cracked or stepping into a space no one expects you to lead.

Chapter 8: The Outsider Advantage

David introduces us to a surprising insight drawn from science competitions like the InnoCentive challenge. In these competitions, organizations post difficult R&D problems to a global network, inviting anyone, not just internal experts to propose a solution.

You’d expect that the most “qualified” participants would dominate. The people with deep experience in the exact domain of the problem. But that’s not what happens.

In fact, the data show that solutions are most often found by people who are outside the domain entirely, but close enough to understand the problem, and different enough to see it with fresh eyes.

For example, a concrete durability problem posed by a construction company was solved by a chemist. A molecular biology question was answered by someone in computer graphics.

David calls this “cognitive lateral thinking.” It’s what happens when people carry mental models from one domain into another and see things the insiders miss.

Why This Is Vital for Tech Professionals

Let’s now apply this to tech careers, especially in AI-heavy or innovation-driven environments.

Many technical professionals carry deep expertise in a specific stack, system, or architecture. They’ve been trained to solve problems with best practices. But in rapidly changing contexts new product categories, cross-functional initiatives, novel AI use cases best practices don’t exist yet.

The insiders often apply what they already know. But what if the problem requires a completely different frame?

That’s where the outsider shines.

If you’re a software engineer who understands organizational psychology, you might design more human-centered AI products.

If you’re a data analyst who dabbles in regulatory law, you might see compliance risks before legal does.

If you’re a systems architect who studies storytelling, you might communicate infrastructure tradeoffs in a way that influences executives. My last job in tech was that of a systems architect in 2007, way before I l had the opportunity to learn storytelling through a series of other roles I took on after that.

Developing such lateral skills aren’t “side hobbies.” They’re strategic advantages. They are the source of your range.

When Not Knowing the Rules Is Exactly the Point

David explains that many breakthroughs happen when people don’t know what they’re not supposed to try.

He gives the example of Gunpei Yokoi, the inventor behind Nintendo’s Game Boy. Yokoi wasn’t the most advanced engineer. In fact, he worked with “lateral technology” that was not only mature but also used inexpensive components. His genius wasn’t in bleeding-edge tech, but in recombining old parts in new ways.

That’s how he helped turn Nintendo from a playing card company into a global gaming empire.

He didn’t break the rules. He just never learned the ones that kept others stuck.

In the Age of AI, Every Domain Is a Mashup

Modern tech work doesn’t live inside neat categories anymore.

• DevOps is blending with security.

• ML is merging with design.

• Product thinking now includes ethics, trust, and regulatory nuance.

• Engineering leaders are expected to understand GTM strategy, pricing, and user behavior.

That means being an outsider in one part of the conversation is inevitable. The question is whether you use that as a constraint or a superpower.

If you’ve ever felt like the odd one in the room too business-oriented for engineering, too technical for marketing, too strategic for ops, then congratulations! That’s your edge.

Practical Strategy: Turn Outsiderness Into Leverage

1. Own your interdisciplinary background

   If you came from a non-traditional path say, economics before engineering, or graphic design before data highlight that. It gives you pattern recognition others don’t have.

2. Volunteer for ambiguous, cross-functional projects

   These are often the ones insiders avoid. But they’re ripe for outsiders to see solutions no one else does. Bring your outside lens to the table.

3. Study problems in unrelated domains

   Find challenges your team is facing and study how other industries solve them. You’ll start to build a library of metaphors, models, and mechanisms that apply across contexts.

4. Don’t wait for permission

   Outsiders don’t always get invited into core decisions. Sometimes you have to show up with a solution first. If you see a blind spot, raise it. If you have an unconventional proposal, pitch it. Respectfully challenge the assumption that “we’ve always done it this way.”

Bottom Line: Your Range is your invitation to what’s next

The next opportunity in your career may not come from what you’ve mastered. It may come from where your strange combination of skills lets you see something others missed.

Being the outsider doesn’t mean you don’t belong. It means you see the game from a new angle.

In a world shaped by rapid change, the outsider is often the innovator. And the person with range is the one who gets to rewrite the rules.

In the next chapter, we’ll go deeper into how seemingly “outdated” tools and experiences can lead to radical innovation when recontextualized creatively.

This chapter takes a curious but powerful turn. David introduces a concept that might sound counterintuitive in a world obsessed with cutting-edge tech and constant upskilling. He argues that old technology, when repurposed laterally, can fuel breakthrough innovation.

It’s not just about chasing what’s new. It’s about seeing what’s useful, even if the world has already moved on.

And for tech professionals navigating the AI age, this chapter has profound implications.

Chapter 9: Lateral Thinking With Withered Technology

The phrase “lateral thinking with withered technology” comes from Gunpei Yokoi, the legendary Nintendo engineer we touched on in the last chapter.

Yokoi didn’t try to build the most powerful hardware. Instead, he embraced outdated components, combined them in unconventional ways, and created iconic products like the Game Boy. While his competitors chased graphics and processing speed, he used cheap, well-understood parts to make games that were simple, fun, and durable.

The Game Boy went on to become a massive global success. Not despite the outdated tech but because of it.

Yokoi’s philosophy was this: when technology matures and stabilizes, it becomes affordable, predictable, and easier to adapt creatively. You can recombine it in new contexts without the risks of bleeding-edge fragility.

That’s lateral thinking. And it works because innovation isn’t always about pushing forward. Sometimes, it’s about looking sideways.

Powering Real World Generative AI Architecture: By Being Boring And Brilliant At The Same Time

In tech careers, it’s easy to become obsessed with “what’s next.” The newest AI model. The latest framework. The next “latest”.

But David’s message, through Yokoi’s story, is simple. You don’t always need to chase the frontier. You can innovate by recombining what already works.

Imagine you’re a senior backend engineer watching your team run out of creative energy. Why?

Everyone around you is chasing the “bleeding edge”: Rust microservices, WASM at the edge, and complex event-driven architectures. On paper, it’s a masterpiece. In practice, the cognitive load is crushing the team. New hires spend weeks just trying to trace a single request through the “event-driven mess,” and the product team hasn’t shipped a meaningful feature in months.

While they hunt for the next shiny framework, you’re looking at your “dated” toolkit: Postgres, Redis, and a battle-tested Monolith.

You realize that creativity isn’t about using the newest tool; it’s about the creative recombination of proven ones. You propose something that sounds almost heretical in a hype-driven office: an internal MVP engine built on a “Boring Technology” stack.

You don’t need a specialized vector database for the new AI feature you know Postgres can handle it with an extension. You don’t need a complex distributed cache for the prototype Redis is already sitting there, ready to go.

The Breakthrough

While the rest of the org is fighting “unknown unknowns” in their experimental stack, your “boring” engine is live in forty-eight hours.

You haven’t just built a prototype; you’ve achieved Operational Maturity overnight. Because you chose a stack where you already know where the “landmines” are buried, you can focus 100% of your creativity on the user experience rather than the infrastructure.

Suddenly, your “outdated” skills are the company’s greatest accelerator. You didn’t win by being the fastest coder; you won by knowing that simplicity is the ultimate sophisticated recombination. You saved the company’s innovation tokens for the product, not the plumbing.

Old Tech + New Context = Strategic Leverage

Lateral thinking with withered technology isn’t about nostalgia. It’s about transference.

You’re using experience with known tools to solve new problems in unfamiliar domains.

And this is where range comes in.

• The frontend developer who knows psychology can build interfaces that anticipate user confusion.

• The data engineer who’s worked in fintech and education can see patterns that a specialist can’t.

• The cloud architect who understands physical logistics can design better infrastructure for distributed systems.

You don’t need the newest thing. You need the right combination of old and new as well as the imagination to apply it laterally.

AI is Built on This Philosophy

Ironically, the entire AI boom is proof of this principle.

The underlying architecture of transformers, which powers GPT and other LLMs, wasn’t invented last year. It came from a 2017 Google paper called “Attention Is All You Need.” And that paper built on decades of earlier work in NLP, probability theory, and gradient descent. If you are interested in this paper, you can find it here on Arxiv.

The magic wasn’t in brand new ideas. It was in recombining known components at the right time, with the right compute infrastructure.

In other words, AI itself is lateral thinking with withered technology.

So if you’re worried that your skills are getting “stale,” ask yourself instead, Where can these tools shine next?

Subscribe now

Practical Strategy: Don’t Discard, Reframe

1. Inventory your “stale” strengths

   Make a list of tools, skills, and mental models you’ve mastered that the market thinks are “old.” For each, ask: “Where might this be useful in a new context?” You might be surprised how many doors open when you change the setting.

2. Recombine before you reinvent

   When solving a new problem, start by asking: “What have I used before that might apply here differently?” Don’t reach for new tools by default. Reach for new angles first.

3. Teach incoming staff the value of the ‘old ways’

   Junior engineers might dismiss technologies you know inside out. Show them how reliability, simplicity, and cost-effectiveness still win in many settings. That turns your knowledge into mentorship and range into leadership.

4. Use constraints as a creativity engine

   Don’t wait for unlimited resources. Challenge yourself to solve problems using “low-power” tech. Old methods. Limited compute. That’s how lateral creativity forms.

Bottom Line: The Past Is a Source of Innovation

You don’t need to abandon everything you’ve learned to stay relevant. In fact, the most future-proof professionals aren’t the ones constantly chasing new tools. They’re the ones who can see possibility where others see obsolescence.

Old skills. Legacy systems. Mature tools. These are not liabilities. In the hands of someone with range, they’re ingredients for strategic breakthroughs.

In the next section, we’ll look at an interesting and somewhat contrarian idea “Fooled by Expertise”, where David shows how even the most experienced professionals can be led astray by overconfidence, and why humility is a key part of building effective leadership in a complex world.

This section discussed one of the most cautionary and relevant chapters for anyone navigating complex systems, especially in tech.

This is where David pulls back the curtain on one of the biggest risks in modern careers: overconfidence born from deep expertise. And in a world where decisions must be made across messy, unpredictable, AI-transformed environments, this blind spot can cost more than credibility. It can cost outcomes, influence, and the chance to lead.

Chapter 10: Fooled by Expertise

This chapter opens with a stark and troubling study: Philip Tetlock’s famous 20-year forecasting experiment. Tetlock collected predictions from hundreds of political and economic experts, asking them to forecast real-world outcomes including wars, elections, market crashes.

The results were damning.

The most famous experts, the ones with the highest credentials and deepest specialization performed no better than random chance. Some were worse. Why? Because the more deeply they specialized, the more confident and committed they became to their own frameworks. They filtered out conflicting data. They ignored ambiguity. They rationalized poor outcomes.

The conclusion? In complex, fast-changing environments, narrow expertise isn’t enough. Worse, it can actively mislead.

And here’s the kicker: the most accurate forecasters weren’t the specialists. They were the generalists, the “foxes” (as Tetlock called them). These were the people who knew a little about a lot and were constantly updating their models, cross-checking their biases, and staying curious.

Where do you see this?

Let’s put this in the context of a senior engineer or data leader in 2026.

You’ve spent the last decade mastering a domain — backend infrastructure, enterprise data architecture, ML deployment. You’ve earned your place as the go-to expert.

But now AI models are becoming black-box co-workers. Regulation is accelerating. Security threats are escalating. User behavior continues to shift faster than ever. Consider this: People don’t “Google” for information anymore, they “ChatGPT” or “Gemini” etc for answers. Subtle but highly nuanced behavior disrupting the whole ecosystem and business models build on Search Engine Optimization. Bringing us back to you as the data leader or the senior engineer we started to talk about, your leadership is suddenly expected to extend beyond your area of technical comfort. And why is it that you are expected to do that? Because emerging technologies create emerging landscapes and emerging multi-domain problems that hardly anyone is an expert to solve for.

In this moment though, overconfidence can become a career-limiting condition.

• You insist a new system can’t scale, because it doesn’t follow traditional sharding best practices.

• You dismiss an AI product idea because it doesn’t fit your model of “real ML.”

• You push back on design trade-offs because they conflict with your engineering instincts, not because you’ve tested outcomes.

And slowly, you become less effective, not more. Because your deep domain expertise starts blinding you to a changing landscape.

The AI Age Demands Humble Leaders

This is one of the central arguments David makes. In wicked environments where rules change, data is incomplete, and feedback is delayed the best decision-makers are humble, adaptable, and multidisciplinary.

In other words, they have range.

They are not the loudest voice in the room. They’re the ones asking questions. Testing assumptions. Borrowing models from other fields. Updating their priors based on new evidence.

In Tetlock’s research, these “foxes” were:

• More likely to acknowledge uncertainty.

• More open to input from outside their domain.

• More likely to change their mind.

• More accurate over time.

If you want to lead in AI, where no one can be an expert in everything, this mindset is non-negotiable.

Real-World Parallel: The 2008 Financial Crisis

David ties the lesson of expert overconfidence to the 2008 crash. Many of the so-called smartest financial minds PhDs, Nobel laureates, quants were completely blindsided by systemic risk. Their models, which had worked in predictable environments, failed in reality.

During my days at B-school, I recall being in a class in 2009 at NYU Stern where I was sitting next to a classmate who had previously graduate from MIT with a degree in engineering. She worked as quant at a leading rating agency in the city. She validated this exact point for me we were hearing in the media - “No industry insider seemed to have seen this coming”. This whole crisis seemed closer to home for us given its recency, its continuing fall out, and more so because we were in a classroom about two miles from the epicenter of this meltdown. (Map below).

But who saw the crisis coming? Often, it was outsiders journalists, generalist investors, or people who cross-referenced multiple fields. People who saw the connections others missed. People who questioned the sacred models.

The analogy to AI-driven tech is clear. Many of the best decisions won’t come from specialists. They’ll come from people who know just enough across multiple domains to spot the disconnects.

Practical Strategy: Adopt a Fox Mindset

1. Regularly update your frameworks

   Don’t cling to the same decision-making playbook forever. Create a habit of reevaluating your assumptions quarterly. Ask, “What has changed in the environment that should change how I think?”

2. Invite contradiction

   Seek out voices that disagree with you including other teams, backgrounds, or functions. Encourage dissent in your meetings. This doesn’t weaken your authority. It strengthens your decisions.

3. Cultivate humility

   Get comfortable saying “I don’t know yet.” It’s not weakness. It signals intelligence in complex environments. Model this for your teams.

4. Study other disciplines

   Learn how strategists, product designers, psychologists, and historians think. You’ll start seeing failure patterns and solution patterns your peers overlook.

5. Test small, iterate fast

   Avoid placing big bets based solely on confidence. Set up small experiments that generate real data. That’s how foxes build confidence not from belief, but from feedback.

Bottom Line: Don’t Let Your Strength Become Your Limitation

In a world where AI is changing the rules faster than we can codify them, being “the expert” is less valuable than being the adaptable thinker.

Yes, your experience matters. But your range: your ability to integrate diverse models, admit uncertainty, and adjust will determine how far you go.

The best leaders in the AI era won’t be the most confident. They’ll be the most curious.

In the next section the cautionary tone develops into signaling of danger which in my opinion is highly relevant for tech professionals as well. David explores why professionals often fail when they refuse to let go of legacy ways of thinking and how this affects real-time decision-making under stress.

If the last chapter warned us about the dangers of overconfidence, this one delivers the natural follow-up: What happens when we cling to the tools, methods, and models we know even when the world changes around us?

David makes a compelling case that failing to drop familiar tools isn’t just inefficient can be deadly. For tech professionals navigating rapid change, that’s not a metaphor. That’s a roadmap for irrelevance, unless we develop the muscle to adapt.

Chapter 11: Learning to Drop Your Familiar Tools

David draws from a tragic and powerful example: firefighting disasters, specifically the 1949 Mann Gulch fire that killed 13 elite smokejumpers. The men who died weren’t untrained or underqualified. In fact, they were some of the best in the business.

So what happened?

When the fire turned unexpectedly, they were instructed to drop their tools and run uphill. But many couldn’t do it. They clung to their heavy equipment including shovels, saws, packs. These tools that had become symbols of identity and control. Tools that made them feel like professionals.

They slowed them down. And they died.

In later incidents, it happened again. Highly trained responders failed to abandon familiar tools, even in life-threatening scenarios. Why? Because tools aren’t just objects. They’re extensions of self. Dropping them felt like dropping identity.

The result? Inflexibility in the face of change. Tragedy in the face of speed.

What This Means for Tech Professionals

In tech, our tools are usually digital. But our emotional attachment to them is just as real.

• The framework you’ve used for years.

• The data model you built from scratch.

• The codebase you’ve maintained like a second brain.

• The process you refined and defended over time.

These aren’t just tools. They’re symbols of mastery. Of credibility. Of who we are as professionals.

Which is exactly why they’re so hard to let go of—especially when the terrain shifts. Especially when AI changes the very assumptions those tools were built on.

Dropping Familiar Tools in the AI Era

Let’s make this painfully real.

Imagine you’re a seasoned DevOps engineer. You’ve spent a decade hand writing YAML files., meticulously hand-crafting CI/CD pipelines and mastering the art of containerization with Kubernetes. You were the “gatekeeper of production.”

Then, the Abstraction Wave hits. Enter Vercel.

New AI-native deployment platforms emerge that don’t just assist you they commoditize your core tasks. They handle auto-scaling, blue-green deployments, and secret management with a single prompt. 80% of your “hard-earned” expertise is suddenly a background process.

The Pivot

You face the classic “Engineer’s Dilemma”: do you fight the tool by pointing out its edge-case failures, or do you treat the tool as a new primitive?

You choose to evolve. You stop being the “plumber” fixing individual leaks and become the Architect of Velocity. You realize that while the AI can deploy the code, it doesn’t understand System Resilience, Compliance, or Cost-Efficiency.You can resist it, dismiss it, argue its limitations. Or you can learn the new system, drop some of your old tools, and evolve into a new kind of builder: someone who enables velocity through orchestration, abstraction, and systems thinking.

Now multiply that across roles:

• The data scientist who lets go of manual feature engineering in favor of AutoML, and pivots toward interpreting models in business context.

• The engineer who drops custom microservice sprawl and embraces platform thinking, improving performance by simplifying instead of scaling.

• The PM who gives up a strict agile cadence in favor of continuous experimentation based on AI feedback loops.

In all of these examples, professionals grow not by clinging, but by releasing.

This Isn’t About Abandonment. It’s About Reframing.

Dropping familiar tools doesn’t mean throwing away your value. It means updating how your value gets expressed.

It’s the database wizard who becomes the real-time observability expert.

The cloud infrastructure builder who becomes the systems reliability strategist.

The brilliant coder who becomes a tech translator for the C-suite.

You’re not starting over. You’re moving up.

Practical Strategy: Build Your Adaptation Reflex

1. Inventory your identity tools

   Ask: “Which tools, methods, or models do I deeply associate with my professional identity?” These are the hardest to drop. And the most important to question.

2. Run “no tool” challenges

   Once a quarter, try solving a common problem without your go-to tool. Force a constraint. It’ll surface blind spots and make space for lateral solutions.

3. Welcome junior insight

   Junior team members often adopt new tools faster because they’re less invested in legacy ones. Learn from their freshness. It’s not just naivety, it’s agility.

4. Decouple ego from method

   Your value isn’t tied to any one tool. It’s tied to your ability to solve real problems. Focus your pride there.

5. Create exit strategies for old methods

   Build a habit of retiring tools, not just accumulating them. What do you no longer need to carry? What’s slowing you down?

Bottom Line: Adaptation Is a Leadership Skill

In high-velocity environments, those who succeed aren’t the ones who hold the tightest grip. They’re the ones who can let go when the moment demands it.

In firefighting, that reflex saves lives. In tech, it saves careers.

Range is what helps you build that reflex. Because when you’ve experimented across domains, roles, and perspectives, you’re less attached to any single tool. You trust your ability to adapt, not your ability to defend.

And that’s what future-ready professionals do. They don’t define themselves by their tools. They define themselves by the problems they’re ready to solve.

In the final section named Deliberate Amateurs, we explore how embracing the amateur mindset: curious, flexible, unburdened by expert baggage can make you not just more innovative, but more influential.

If this helped you reframe your thinking about your career development please consider sharing with others. (And, feel free to post it to your LinkedIn network).

More content like this in the future. Stay tuned.

Subscribe now

If you missed the previous posts. They can be found here:

Part I is here.

Part II is here.

Part III is here.

If you missed the previous opening posts on this topic:

Part I is here.

Part II is here.

Continuing…

Chapter 4 — Learning, Fast and Slow

In this chapter, David dismantles another sacred cow of conventional career advice: the idea that fast learning is always better learning.

In schools, bootcamps, and even professional development programs, we’re conditioned to seek efficiency. The faster you acquire a skill, the smarter you must be. The more fluently you can recall, repeat, and replicate, the more advanced you’re perceived to be.

But as David shows, there’s a massive difference between fluency in the moment and flexibility over time. And surprisingly, it’s the struggler, not the natural, who often comes out ahead.

The Interleaving Study: Slow Learners, Fast Results

David cites a fascinating experiment with two groups of students learning to identify types of paintings. One group studied paintings by artist A, then artist B, then C. All in “blocked” sessions. The second group studied the same paintings, but mixed them up. This is a technique called interleaving.

The results were revealing.

During training, the blocked group outperformed the interleaved group by a significant margin. They “learned faster.” But when tested later, when the task required transferring knowledge, the interleaved group crushed it.

Why? Because the struggle of interleaving, of comparing, contrasting, and making distinctions created deeper encoding. It felt harder, but it worked better.

David’s conclusion? “Desirable difficulty” the kind that makes learning slower actually leads to more robust performance when it matters most.

Where This Shows Up in Tech Careers

Let’s translate that to your career.

The professional equivalent of blocked learning is what most engineers and analysts do early in their roles: focus on one problem domain, one language, one layer of the tech stack. The work becomes fluent. You can solve things quickly, almost without thinking.

It feels like mastery.

But then something changes: business priorities shifts, your tools evolve, or as we are learning AI is about to automate part of your workflow, if not already. Suddenly, you’re asked to solve a new kind of problem: unfamiliar data, undefined metrics, a new ‘wicked environment’ requiring tradeoffs to be made for design and build that will change business outcomes.

So what happens to the people who struggled more in the early days who rotated through different tools, took on seemingly weird projects no one was willing to touch, or had to explain their work to non-technical audiences? They are the ones primed to make confident decisions in ambiguity. Because they learned how to learn.

The speed of your early fluency is not the same as your readiness for complexity.

AI Makes This Even More Critical

Let’s put this in the context of AI.

AI accelerates syntactic fluency. It helps you autocomplete code, translate frameworks, even write documentation. This is the kind of speed that flatters the fast learner.

But AI can’t do:

• Model-to-business mapping

• Organizational prioritization

• Systems thinking across departments

• Reconciliation and resolution between independently developed roadmap and architecture

• Human intuition in unknown domains

That’s where the interleavers, i.e. the slow, struggling learners win.

Because they weren’t just memorizing patterns. They were building mental models that not only become adaptable to breaking down complexity but scales with it.

Why This Matters in a Wicked World

Wicked environments don’t reward people who are masters of tools. They reward people who can transfer principles from one domain to another.

When the problem is fuzzy, the systems have shaky foundation, you have multiple conflicting priorities - you need more than fluent system builders. You need flexible thinkers.

So if you’ve felt like your learning journey has been messy and if you’ve had to figure things out slowly, through failure, across different stacks or domains, then, that wasn’t wasted time. That was your real edge being built.

Practical Strategy: How to Train Like a Generalist

1. Practice “interleaved” learning in your career

   Don’t optimize for staying in one domain just because it’s comfortable. Mix projects. Pair with someone in a different role. Explain your work to people outside your team. Each friction point builds cognitive flexibility.

2. Do things the hard way (on purpose)

   Every once in a while, write code without your favorite helper tools. Build a model from scratch without AutoML. Recreate a report without a template. Struggle intentionally. That’s where the learning lives.

3. Debrief your decisions, not just your results

   The fast learner just celebrates that something worked. The flexible learner asks: “Why did that work? Would it work here again? What was different?” That reflection is what trains adaptive judgment.

4. Teach what you learn

   Teaching forces you to translate deep understanding into clear explanations. That process slows down learning and makes it permanent.

Bottom Line: Fast is Fragile. Flexible is Forever.

In a world where machines will outperform humans in speed, efficiency, and recall our real advantage isn’t fluency. It’s range-informed flexibility.

The best careers aren’t built by those who sprinted to early mastery. They’re built by those who got their hands dirty across contexts and turned struggle into strength.

Chapter 5: Thinking Outside Experience

This is the chapter where David starts to seriously dismantle the idea that expertise comes only from repetition inside a single domain. In fact, he flips that logic completely. Instead of asking, “How deep is your experience?”, he suggests we ask a more powerful question. “Can you think beyond your experience?”

That’s a different kind of intelligence. And it’s exactly the kind that becomes invaluable in tech careers navigating wicked, unpredictable change.

The Firefighter Analogy That Fell Apart

David opens this chapter by revisiting a famous concept from psychology. It’s the idea that experts can make incredibly fast decisions because they have built up thousands of stored patterns in their brain. The classic example is the firefighter who senses that a building is about to collapse, even though no visible clues are present. It’s intuition, born from experience.

This is the gold standard in many fields. In fact, we tend to revere that kind of “instinctive expertise” in senior engineers, senior staff-level ICs, and technical architects. We assume that the more time someone has spent in a specific domain, the more likely they are to know what to do when a crisis hits.

But David then asks a harder question. What happens when the environment changes? What if you’re suddenly facing a different kind of fire?

He brings up the case of the Columbia space shuttle disaster. In this case, the engineers with the most NASA experience were the ones who insisted everything was fine. They had seen foam strikes before, and nothing catastrophic had ever happened. So they treated this one like all the others.

Except, this time, the foam strike did cause fatal damage. And the people who had the hardest time seeing the risk were the ones with the most direct experience. Their deep familiarity blinded them to a new kind of danger.

Experience Can Be a Liability in Machine Learning Too

If you’ve spent years mastering classical machine learning, you’ve likely developed a strong intuition for how to clean data, engineer features, tune hyperparameters, and validate models through careful experimentation. You’ve earned your credibility by building robust, explainable, well-evaluated systems often for narrow business use cases.

Sine the advent of LLMs, the terrain has shifted.

Use cases now require building and deploying LLMs and working with foundation models. You’re no longer training models from scratch: you’re prompting them, fine-tuning them, or chaining their outputs. Feature importance is not a topic of discussion anymore in this context. There is no hyper parameter tuning. You are designing and experimenting with prompts. Evaluation is no longer a simple confusion matrix as you would use in a classification problem, it’s about user behavior, evaluating open-ended quality metrics on user expectations and experience, as well as controlling and evaluating for all the risks a LLM poses.

Your traditional ML instincts grounded in structured model development process (CRISP-DM / SEMMA), precision, and explainability appears to be “different worldly” for a data scientist. You may try to slow things down, asking questions about where do we reintroduce classical rigor, or question whether this is “real” machine learning.

But that instinct, however valid in the past, might be holding you back now.

Because generative AI is a different paradigm, not just a new toolkit. And applying your hard-earned ML patterns to it without updating your lens can lead to misalignment.

This is how experience even deep technical experience becomes a liability.

Not because it’s wrong, but because it’s out of context for classically trained statisticians, ML engineers or data scientists.

So Where Do the Best Ideas Actually Come From?

In the book, David gives the example of scientists who made breakthroughs outside their primary domains. He cites a study that showed researchers who solved complex innovation problems were more likely to do so when their domain expertise was only loosely related to the problem.

They weren’t so close that they relied on familiar patterns. But they weren’t so far that they couldn’t grasp the problem at all.

That sweet spot, known as far transfer, is where innovation lives.

In practical terms, that means the best solutions to your hardest problems might come from someone who doesn’t “specialize” in that exact space. They might work in design. In ops. In legal. Or they might be you, if you’ve taken time to build perspective across disciplines.

Why This Matters in an AI World

As AI starts to become embedded in every workflow, the boundaries between disciplines is likely to dissolve or at least introduce a AI-Human integrated layer..

An engineer today might need to understand not just how models work, but how people trust them. A product manager might need to reason about ethics, safety, infrastructure cost. A technical lead might need to navigate product strategy, user privacy, and regulatory compliance. Consider this: the job title of “AI Engineer” was practically non-existent prior to advent of LLMs becoming mainstream.

In other words, you don’t just need more experience. You need the right kind of experience. That means varied, cross-disciplinary, and intentionally adjacent.

And it also means you need to practice thinking outside of it. Not just “What do I know that applies here?”, but “What is just outside my experience that could inform this?”

That’s what makes a strategist. That’s what makes someone future-proof.

Practical Strategy: Train for Far Transfer

1. Read outside your domain

   If you’re a data person, read product design case studies. If you’re in backend engineering, explore behavioral psychology. The more frameworks you can draw from, the more novel connections you can make.

2. Run solution sprints with outsiders

   Bring in someone from an unrelated team when solving a tough problem. Not for execution help, but to ask questions that break your usual framing.

3. Challenge your mental models

   Ask yourself, “If I had to explain this to someone in marketing, how would I do it?” This forces you to abstract your knowledge and look for new metaphors.

4. Adopt a second lens

   Pick a second professional identity to cultivate. Are you an engineer who also studies business models? A data analyst who reads philosophy? That second lens gives you clarity no specialist can replicate. Personally, I have spent a good part of a decade adopting a second lens myself. It definitely continues to serve me well.

Bottom Line: Mastery Is a Ceiling. Range Is a Bridge.

In a world where everything is changing from how we build systems to how we trust machines, the person who can think outside their experience will always have an edge over the one who only deepened theirs.

The next leap in your career won’t come from going deeper. It’ll come from seeing sideways, connecting dots no one else noticed, and pulling in lessons from other domains.

Chapter 6: The Trouble With Too Much Grit

We’ve been taught that grit is a defining trait of success. It’s the quality behind “winners never quit and quitters never win.” It’s the force that powers you through code reviews, late-night incident responses (yes, I also had a job two decades ago that involved both of these type of roles), years of climbing the technical ranks.

But David asks a more uncomfortable question: What if grit keeps you stuck?

In this chapter, David makes an important pivot. He challenges not just how we learn, but how we persist. While grit, the celebrated ability to push through challenges and ‘never quit’ has become a professional virtue in popular self development literature, David offers a counter point of view that talks about the dark side of sticking to a path too long, especially when the world around you is shifting.

The Escalation of Commitment

David shares the story of elite students who enter prestigious career tracks such as pre-med, law, finance and feel pressure to persist, even when it becomes obvious the path no longer fits them. These individuals aren’t failing. In fact, they’re often succeeding on paper. But they’re also stuck in what psychologists call the escalation of commitment. It is a tendency to continue investing in a decision because of the time and effort already sunk into it.

It’s not just that they fear quitting. It’s that they’ve spent so long building an identity around being gritty, that switching feels like personal failure. How many people do you know professionally who seems like this persona? Anyone?

David shows that many of the most innovative thinkers and leaders aren’t the grittiest in the traditional sense. They’re the ones who know when to walk away from the wrong path, even if it means disappointing others or starting over.

In other words, it’s not just perseverance that matters. It’s strategic quitting. (I also did this in 2007 - I wrote about this in part 1 here if you missed it)

Why AI Changes the Risk Equation

In an AI-transformed future, domain development lifespan will shrink.

• Tools that once took years to learn can now be semi-automated.

• Categories of technical work are being co-piloted by … you guessed it copilot .. no pun intended (Github etc). Let’s not yet go to the topic of vibe coding. I have a point of view on that I will cover in future posts.

• What was once construed a “career moat” is now a white space for a bright eyed AI startup founder.

If you define grit as loyalty to a specific technical skill, you’re building permanence on sand.

Instead, your resilience needs to come from range: the ability to move, adapt, reframe, and re-engage in new contexts without fear of leaving your sunk costs behind.

Real-World Parallel: Andre Agassi vs. Roger Federer

Though discussed earlier, this is where the Federer analogy from Chapter 1 deepens. Federer didn’t lock himself into tennis at age four. He tried basketball, handball, soccer. He switched late and dominated.

Contrast that with the professional identity crisis Andre Agassi describes in his memoir Open, where early over-specialization left him burned out, struggling, and without a sense of self outside of the sport.

In tech, this happens all the time. People specialize so early and so deeply that when change comes and it always does they have no transferable story, no adjacent skills, and no appetite to start over.

Practical Strategy: Redefining Grit in a Dynamic World

1. Run periodic “career audits”

   Ask yourself: “If I were starting from scratch today, would I still choose this path?” If the answer is no, don’t ignore it. You don’t need to quit tomorrow, but start sampling immediately.

2. De-risk switching by diversifying

   Explore adjacent domains before you need them. Take on stretch projects. Join cross-functional initiatives. Build the escape ramps before the current lane closes.

3. Change your identity from ‘expert’ to ‘explorer’

   When people ask what you do, don’t box yourself into a title. Frame yourself as someone who solves complex problems across contexts. This creates space to evolve without shame. My personal philosophy on this is “Titles don’t define who you are”. Because that is very self-limiting.

4. Measure progress by range, not just role

   Promotions are one form of progress. But so is the ability to navigate uncertainty, influence across domains, and pivot with confidence. Track that as your growth metric.

Bottom Line: Smart Quitters Win

The most resilient tech professionals won’t be the ones who went the deepest. They’ll be the ones who knew when depth became a dead end, and had the courage to move laterally or even diagonally to stay relevant.

Letting go of a skill, title, or path doesn’t mean you gave up. It means you saw the future coming and moved.

Chapter 7: Flirting With Your Possible Selves

In this chapter, David introduces the concept of “possible selves”, a term from psychologist Hazel Markus. It describes the mental versions of ourselves we carry in our heads. The different careers, different lives, or different roles we could become. These are often undefined, rough sketches, not clear goals. But they serve a crucial purpose. They help us try, test, and refine who we really are.

David argues that success doesn’t come from identifying your “passion” early and chasing it relentlessly. Instead, it comes from actively testing different versions of yourself, seeing what fits, and letting experience inform identity, not the other way around.

He draws on the story of Frances Hesselbein, a woman who didn’t step into a leadership role until her 50s. She tried on multiple professional identities across her life: teacher, volunteer, community member, before eventually becoming CEO of the Girl Scouts and one of the most influential voices in leadership. Hesselbein didn’t plan her way into greatness. She sampled her way there.

Why This Matters in Tech

In tech, we are obsessed with clarity. Job ladders. Career paths. Titles. Promotions. Specializations. But clarity can easily harden into rigidity. And before long, the question “What do you do?” becomes a cage, not a launchpad. Again, self-limiting.

Here’s how it often plays out:

Say, you start as a marketing analyst. You build performant dashboards, analyze experimentation results, apply attribution models, know marketing KPIs inside out. You get known for it and rewarded for it. You become “the marketing analytics person.” Then someone asks, “Have you ever thought about moving into marketing strategy because you understand marketing so well” Or “You have a knack for storytelling, why not join marketing comms”

You respond, “That’s not really my thing.” But … how do you know?

That’s the point of trying on possible selves.

You don’t know which version of your career might fit or where your hidden strengths lie until you give yourself permission to test the edges.

This isn’t about abandoning your data skills.

It’s about using them as a foundation to explore what else you might be good at that is in the next adjacent space.

Because in a world where AI can generate queries and run analyses, it’s not just about who’s the best analyst, it’s about who has the range to connect data to influence, decisions, and strategy.

And you can’t build that kind of range if you never leave the role you’ve been told you’re good at.

The Professional Power of Trying Things On

David’s research shows that people who explore broadly, who take detours, pivots, lateral moves, or even temporary regressions are more likely to land in careers that match both their strengths and their values.

Why? Because your sense of identity emerges from action, not introspection.

Trying out new domains, functions, or roles gives you feedback loops you can’t get from thinking alone. It tells you how your skills translate. It reveals what kind of work energizes you. It shows you which environments reward your instincts.

And in a world being reshaped by AI, this becomes your survival tool.

What It Looks Like in a Tech Career

Let’s say you’re a data scientist with solid modeling skills. But you’re curious about product strategy. You’ve never done it. You’re not sure if it’s for you. But part of you wonders, “Could I be good at that?”

You don’t need to quit your job or get an MBA.

You could:

• Sit in on product roadmap meetings.

• Shadow a PM for two sprints.

• Take on a project where you translate data insights into feature prioritization.

• Run a small experiment where you pitch a data-driven initiative to a cross-functional team.

In doing that, you try on a possible self. i.e. the new strategic PM, the hybrid lead, the internal consultant. You don’t have to marry that identity. But you get to ask, “Does this feel like me?”

Multiply that process over years, and you’ll have a wide portfolio of tested identities. Which means when the time comes to pivot either because of interest or disruption you’ll know exactly where you fit next.

In an AI-Transformed World, Optionality Is Everything

As AI does more of the repeatable tasks in technical roles, affording yourselves the optionality to move between value-creating roles becomes the real “career” moat.

The engineer who can hold product conversations.

The analyst who can evolve into an leader.

The technical IC who can step into strategy without losing credibility.

These are not linear transitions. They’re the result of possibility sampling of having tested multiple selves in lower-stakes settings, so you’re ready when opportunity arrives.

Practical Strategy: Expand Your Range of Selves

1. Run career “micro-experiments”

   Choose one identity you’re curious about (e.g. technical pre-sales, solution architect, team lead). Set a 90-day goal to explore it lightly. Shadow someone. Take on a side project. Reflect on how it feels.

2. Track energy, not just skill

   When you try new things, don’t just assess whether you’re good at them. Ask: Did this energize me? The best roles lie at the intersection of ability and alignment.

3. Decouple identity from title

   Don’t limit your professional identity to your job description. Try introducing yourself in broader terms. For example, “I help organizations turn messy data into clear decisions.” That gives you flexibility to try new expressions of the same value.

4. Stay open to unlikely paths

   Frances Hesselbein didn’t plan to lead a major organization. Her leadership identity emerged from service. Your best opportunities may not come from plans. They may come from patterns that only make sense in retrospect.

Bottom Line: Test, Don’t Assume

The most successful and adaptive tech professionals aren’t the ones who picked the perfect ladder. They’re the ones who explored enough paths to build a multidimensional map of themselves.

If you feel unclear about where your career is going, that’s not failure. That’s signal. It means you’re still exploring, still testing, still building range.

The next big idea covered in the book is based on The Outsider Advantage, where we’ll explore how people from outside a domain often make the biggest breakthroughs, and why that matters when you’re thinking about your next professional leap.

Until next time

Vijay

If this helped you reframe your thinking about your career development please consider sharing with others. (And, feel free to post it to your LinkedIn network).

More content like this in the future. Stay tuned.

Share The Fractal

If you missed the first part of this - please go back here so you have the full context of what this is about, and then come back to this article.

Chapter 1: The Cult of the Head Start

In almost every industry, we reward people who specialize early. If someone “knew they wanted to be a doctor since age 8” or “wrote their first line of code at 10,” we treat it as a badge of honor. In tech, it’s no different. The faster you declare your domain the more credibility you’re often given. It’s the gospel of the head start: pick a lane early, go deep in that area, and enjoy the benefits of becoming invaluable specialist.

But what if that advice, while occasionally true, is often dangerously incomplete? Especially now.

David opens Range by challenging this cultural narrative head-on. He does it not by critiquing it theoretically, but by showing us two real-world cases: Tiger Woods and Roger Federer. These aren’t just sports stories, they’re opposing archetypes of how excellence can emerge.

Tiger vs. Federer: Two Paths to Greatness

Tiger Woods is the canonical example of early specialization. At just seven months old, he mimicked his father’s golf swing. By two, he was on national TV. By three, he was shooting under par. His life was golf and nothing else. This is the prototypical tech professionals we’ve come to admire and ask others to emulate.

Roger Federer, on the other hand, couldn’t have had a more different trajectory. As a kid, he played soccer, basketball, skateboarding, wrestling, skiing and yes, some tennis. His parents encouraged sampling, not specialization. He didn’t take tennis seriously until his mid-teens. No intensive coaching. No travel circuit. No deep athletic seasoning. Just wide play. And yet, Federer would go on to become one of the greatest tennis players of all time.

These two athletes both reached the pinnacle of their fields, but through radically different developmental paths.

And this is where David makes his key point: while Tiger’s path works in a “kind” environment, Federer’s path is better suited to a “wicked” environment.

Kind vs. Wicked Environments: What’s the Difference?

A “kind” learning environment is predictable, rules-based, and has near-immediate feedback. Think of golf, chess, classical music. Any field where repetition leads directly to mastery. You swing, you miss, you adjust. It’s a quick feedback loop. Specialization here is rewarded early and often.

But most things in life including most careers, especially in tech do not operate this way.

A “wicked” environment is one where the rules aren’t clear, the feedback is slow or inaccurate, and the situation evolves constantly. Think of entrepreneurship, product management, leading engineering teams, navigating cross-functional influencing where the success signals are noisy and messy.

In these environments, the habits and instincts you build from early specialization can actually work against you because the patterns don’t repeat predictably.

That’s where generalists like Federer have the edge: they learn to improvise, adapt, and reason through ambiguity. They learn how to learn, not just how to execute a playbook. Their edge isn’t technical precision, it’s strategic fluency. You learn to not only embrace ambiguity, but anticipate ambiguity. Those are the norms of a wicked environment.

What This Means for Tech Careers

Let’s talk about “Tiger Path” professionals in tech. You know the ones. They picked a specialty early, let’s say Ops/Engineering. They became the go-to infra person, solved every scaling problem, fine-tuned the performance of workloads.

Early on, this pays off. They are indispensable.

But then, the context shifts.

Suddenly, the team needs someone to decide whether to refactor the architecture to support a new business unit. Or whether to sunset a platform and migrate to cloud-native tooling. Or how to frame the value of the team’s work to leadership. The environment becomes wicked. The problem space is no longer technical, it’s systemic, strategic, and cross-functional.

And here’s the hard truth: if you’ve only lived inside the “kind” world of technical execution, your instincts may not translate. In fact, they might actively limit you.

Now imagine someone on a “Federer Path.” Early in their career, they bounced between frontend, data modeling, and dev tooling. They explored product conversations, joined a sprint with the sales enablement team, worked on both successful and failed launches. Their path looked non-linear. Maybe even slower. But it built one thing early: range.

And once the environment turns wicked, range becomes leverage.

Why This Matters Now in the AI Era

AI is systematically absorbing many of the kind-environment tasks in tech: writing boilerplate code, debugging syntax errors, generating SQL queries, even suggesting infrastructure diagrams. It rewards those with execution depth but commodifies that depth quickly.

The last thing you want is to be narrowly specialized in a task AI can now do faster than you.

But wicked problems? The fuzzy, strategic, human ones? Those still require range: how to negotiate tradeoffs, assess risk, integrate product-market context, and work across disciplines.

We’re moving from a world where being a “Tiger” gave you a ten-year head start to one where being a “Federer” makes you future-proof.

Practical Strategy: What You Can Do

1. Audit your “environment”:

   Are most of your problems kind (repetitive, with clear feedback) or wicked (ambiguous, cross-functional, change management)? If the latter, are you equipped with enough range to navigate them?

2. Shift from efficiency to flexibility:

   Don’t optimize for faster delivery. Optimize for broader understanding. Take on projects with uncertain outcomes. Volunteer to work on initiatives outside your core stack.

3. Delay final specialization:

   Just because you’re successful in your niche now doesn’t mean you’ve found your long-term “fit.” Allow yourself a career sampling period, no matter how many years of experience are behind you. Think about the active years of experience in front of you where you can make meaningful contribution.

Bottom line: Early specialization makes you faster.

Range makes you adaptable.

And in a world that’s changing faster than your domain can keep up, adaptability always wins.

Chapter 2: How the Wicked World Was Made

In Chapter 2, David dives deeper into the distinction between “kind” and “wicked” learning environments. It’s the concept introduced in Chapter 1 but now rigorously defined. And he starts by examining one of the most high-stakes, feedback-critical professions out there: medicine.

The setup is simple: how do people learn to make decisions? What makes someone an expert? And what happens when the environment they’re operating in doesn’t give them the kind of feedback needed to learn from their experience?

The implications, as we’ll see, go far beyond medicine. It directly impacts how you, as a tech professional, navigate your career in an AI-dominated future.

The Cardiac Diagnosis Story: When Experience Doesn’t Help

One of the most memorable studies David explores involves emergency room physicians diagnosing chest pain.

The researchers discovered something strange: experienced ER doctors often performed worse than novices when it came to accurately diagnosing heart attacks. Their error rates were higher, even though they had years of practice. Why?

Because they were relying on instinct built in a wicked learning environment — one where feedback is slow, incomplete, and ambiguous.

When patients came in with chest pain and were misdiagnosed, but then went to another hospital or died later at home, the original doctor often never knew. The learning loop was broken. And over time, the wrong mental models solidified.

Contrast that with a kind learning environment, like chess or firefighting. You make a move, the result is immediate, and the pattern reinforces or corrects your instincts quickly. These are domains where experience reliably builds expertise.

But medicine — like much of modern life — is different. It’s complex. Multivariable. Pattern-defying. That’s wickedness in action.

Wickedness in Tech: You’re Swimming in It

Now let’s bring this home.

You might think, “Well, I’m not diagnosing heart attacks. I’m building systems.” But think more carefully about how you develop judgment at work.

Let’s say you advocate for redesigning a system to support a growing customer segment. It requires capital. You are in a capital constrained environment. No investments occur. No redesign or re-engineering takes place. The highly anticipated customer segment growth does not come to last, because the systems were never scaled to meet the levels of satisfaction through the customer experience delivered based on legacy implementation. A year later, customer churn spikes in that segment, but no one traces it back to that architectural tradeoff. If you are not in a double digit growth industry (Silicon Valley I am not looking at you), then this will sound familiar.

These are wicked feedback loops. Outcomes are decoupled from decisions. Signals are distorted or delayed. You might be right and still look wrong. Or wrong and still look like a star.

And the biggest trap? You start believing your own faulty patterns.

Like those ER physicians, you begin to trust gut feel that’s been trained in a messy, ambiguous environment. You overfit to patterns that don’t generalize. You optimize for what gets rewarded locally, not what’s right systemically.

AI Makes the Environment Even More Wicked

Here’s where it gets even more interesting and relevant to your career.

AI is removing kind-environment tasks at scale. Repetitive queries, straightforward scripts, basic code translation many of these are now accelerated or being worked on to be replaced. That means what remains on your plate is disproportionately wicked.

You are no longer going to be being asked to “code the thing.” You are going to be asked to:

• Decide which thing is worth coding

• Translate ambiguous business goals into system architecture

• Influence a cross-functional group with competing priorities

• Make calls under uncertainty, where tradeoffs are murky and outcomes are months away

This is wicked territory. And if you’re used to feedback-rich, rule-based environments, this new terrain can feel disorienting.

But it’s also where the opportunity lies, because most of us are still trying to operate like it’s a kind environment. Which means those who ask the question “Ok, what do I need to prepare myself to be ready for a wicked environment” will develop an edge.

Why Range Helps You Navigate Wickedness

This is where David’s thesis and your career strategy start to converge quite powerfully.

In wicked environments, what works isn’t deep repetition. It’s broad pattern recognition.

The doctors who performed best at diagnosing cardiac events? They didn’t have more years of ER experience. They had exposure to multiple domains of medicine. They had developed conceptual models, not rote responses.

In your career, that means the more domains you’ve touched product, ops, sales support, finance ops, systems design, market / social research, organizational design (to name a few), the better you’ll perform when the rules are unclear and feedback is messy.

You’ll see analogies others miss. You’ll avoid overfitting to short-term patterns. You’ll frame decisions in ways that account for ambiguity, not in spite of it.

Practical Strategy: Building a Wicked-Proof Career

1. Shift from solution muscle to sensing muscle

   Stop asking, “What’s the best tool for this problem?” Start asking, “How is this problem evolving, and who else is affected by it?”

2. Create your own feedback loops

   Don’t wait for outcomes to validate your decisions. Build a practice of retrospective inference: six months after a project, ask, “What did I believe? What happened? What does that teach me?”

3. Learn across functions, not just within your stack

   Read about decision-making. Study organizational behavior. Understand how incentives shape architecture choices. Range lives in the overlaps.

4. Say yes to ambiguous projects

   Join the working group with unclear outcomes. Take the product strategy sprint with no defined metrics. Build the muscle of navigating fog. That’s where range thrives.

Bottom line:

Kind environments reward mastery. Wicked environments reward metacognition. It is the ability to think about how you’re thinking.

As AI clears out kind problems, your real value lies in how well you navigate the rest. And that means becoming the kind of person who doesn’t just solve problems, but reframes them entirely.

Chapter 3: When Less of the Same Is More

After laying the groundwork for how wicked environments require a different kind of thinking, David now turns to a question that challenges one of the most accepted beliefs in professional development: Does early, narrow focus actually help you succeed faster?

The answer, it turns out, is not only no, but in many fields, the opposite is true.

This chapter is where David begins to dismantle the myth that linear paths and early clarity about your goals are the fastest (or even most efficient) ways to succeed. In fact, he introduces one of the most powerful career levers most professionals overlook: sampling.

The Army Officer Study: Fast Doesn’t Mean Forward

One of the more unexpected stories David shares is a large-scale study on U.S. Army officers. The study followed their career trajectories and evaluated their performance across time.

It found that officers who sampled more assignments early in their careers — rotating through different posts, roles, and units were initially slower to advance. They didn’t shoot up the ranks like their peers who stuck to one track.

But here’s what happened later: the samplers surpassed their more narrowly focused peers. They led more effectively, scored higher on assessments of judgment and strategic decision-making, and ultimately rose to more senior ranks.

Early specialists got the head start. But generalists won the race.

Why? Because by moving through different roles, the samplers developed transferable skills, pattern recognition, and a wider lens for assessing problems. When the complexity of leadership kicked in, they had the range to handle it. Their early “slowness” was an investment in long-term versatility.

What Tech Professionals Get Wrong About Sampling

Let’s translate this to a tech career.

Early-career developers, analysts, or engineers are often told: “Pick your specialization fast. Go deep. Own a vertical. Make yourself indispensable.”

That sounds great until the context changes. Which it always does.

And Then Came AI…

Here’s where it gets critical in today’s landscape.

AI is reducing the value of ultra-specialized execution. Writing optimized code, querying large datasets, even deploying infrastructure much of that is now being co-piloted, if not outright replaced, by a machine.

But what AI can’t do at least not yet is bridge disciplines. It can’t integrate product context with data limitations and engineering tradeoffs. It can’t negotiate incentives between teams. It can’t hold a mental model that spans five loosely coupled systems and predict where the next problem will appear.

That kind of thinking requires range.

And range comes from sampling.

A gig is not forever, so why not adopt a “I am going to sample this job”

Let’s acknowledge something. Sampling doesn’t always feel strategic while you’re doing it. Just know that hiring managers don’t think anyone they hire are going to stick with the job you take forever. Sooner or later high performers seek the next challenge. Be deliberate about it. And if you so feel, share it. And, of course, there is always a right way to frame it.

Taking a lateral move from ML to DevOps, or rotating into a data governance project, or shadowing a product manager for a quarter, none of those feel like the “next big thing.” Especially when others in your peer network seem to be making vertical ascension in their careers within their specialization.

You worry: Am I falling behind? Am I diluting my expertise?

But if David’s research shows anything, it’s this: non-linear progress is not a weakness. It’s exactly what builds the judgment needed in wicked environments.

And in AI-driven tech, where everything is fluid, judgment beats optimization every time.

Practical Strategy: Designing Your Sampling Phase

1. Map the adjacency graph

   If your core skill is backend engineering, what’s one adjacent domain that’s tightly coupled but not your core? API design? Frontend integration? Dev tooling? Sampling doesn’t mean abandoning, it means expanding.

2. Declare a “range window”

   Set a 6–12 month intentional sampling period. Don’t drift. Pick areas to explore (e.g. “I want to understand business metrics and product prioritization”) and look for projects that expose you to those skills.

3. Use side bets wisely

   Sampling doesn’t have to happen only within your full-time role. Contribute to internal tools, mentor outside your domain, build a tool that solves a non-engineering problem. These side bets become strategic range builders.

4. Reframe your story

   When asked about your career trajectory, don’t apologize for variety. Position it as strategic versatility. Say: “I intentionally explored multiple domains early to better navigate ambiguity and make higher-leverage decisions.” Why? Because ultimately you own your career. There is agency in choosing what you want to do with your career, so why not be intentional about it. And also own what story you want to tell.

Final Thought: You’re Not Late, You’re Building Optionality

In tech, we treat the person who climbed the ladder fastest as the gold standard.

But as David shows through military officers, scientists, athletes, and musicians, the people who climb the highest often didn’t start on the ladder at all. They wandered, explored, cross-trained and then took off with more force and better direction than anyone else.

If your path looks messy, non-linear, or a little slow, that might be the best indicator of future readiness in a world where AI will remove the straight lined paths.

In next part of this series we’ll begin looking at the next big idea Epstein provides:

Learning, Fast and Slow, where we’ll look at why the people who seem to struggle more during learning often outperform in the long term and how embracing inefficient learning can give you the edge in a fast-moving industry.

Until next time,

Vijay

If this helped you reframe your thinking about your career development please consider sharing with others. (And, feel free to post it to your LinkedIn network). More content like this in the future. Stay tuned.

Share The Fractal

It was winter of 2007. At that point, I had spent almost a decade of my career building and designing tech and data on open source technologies. I decided to take a bold and partly seemingly foolish 180 degree turn into a business role as a finance analyst with the ‘feeling’ that it was the right thing to do as a next step in my career, even if it came at the expense of giving up a promising career growth in tech. I worked at one of the largest global companies in the world at their HQ at that time. With a highly supportive management sponsoring my career pursuit, we concluded this transition from tech to finance would be a great next step. It took me from being a desk-bound architect who only ever was among deep tech audience to working with execs who ran P&Ls - something I was not exactly prepared for.

Day 1 on a new job typically brings excitement and a spring in ones’s step, that unfortunately was not quite how it went down for me. My first meeting of my first day was an ‘financial operating review’. On the slide projected on the conference room screen was a deck full of numbers with bare minimum commentary. I was surrounded by finance undergraduates, with most of my colleagues about 5 to 10 years younger than me (some right out of college). Everyone felt poised, calm, confident and provided their ‘insights’ - none of which made any sense to me. Thirty minutes into the meeting, I started developing an uneasy feeling, starting with the thought that I was in the wrong room. Those thoughts started to become heavier with the weight of discomfort, and a terrible achy feeling that I just made a seriously wrong career move. After all, it was becoming clear that I was the least qualified person on that team, because I had no background in finance. I had never taken an accounting class ever ( a pre-requisite to even have that job). Granted, I was getting into B-school that spring, but my accounting and finance classes were not until 6 months in the fall. I was formally an engineering undergraduate and really was a tech-bro.

The next few days I continued to churn through buyer’s remorse. I stayed put. I told myself, ‘I am going to give it a year’. Like anything else, the initial shock of ‘I made a mistake’ came to pass as I started to become resourceful and began learning on the job as well as building my network of experts I could go to when I needed help.

The first year in that job was possibly the lowest ‘productive’ year of my career if productivity was to me measured both in some sense of ‘throughput’ for my employer and personal sense of accomplishment - something you could be proud to claim. I couldn’t escape the feeling that I had hit a new trough.

Many years later I moved onto to do other functional and analytical roles after “grinding” it out and learning everything I possibly could in that job including corporate financial planning, corporate strategy, investment analysis, expense modeling, financial modeling, and, a solid understanding of accounting (I’d be remiss if don’t include accounting standards), as well as the essence of being able to understand and learning to articulate ‘what is the story here’ to communicate with senior executives whose P&Ls I was responsible to manage.

Fast forward more than a decade and a half now, I can unequivocally say that was the most transformative career switch I ever made. The first principles of critical thinking, analytical skills, story telling all came out of that job which has continued to manifest in many of my roles after that point on and allowed me to grow professionally.

This personal journey, which seemed like a career-threatening mistake at first, turned out to be a masterclass in what author David Epstein would later articulate in his groundbreaking book “Range: Why Generalists Triumph in a Specialized World.” Published in 2019, this book arrived at a crucial moment when tech industry is at the epicenter of an AI led transformations across industries that’s reshaping how we think about jobs, expertise, careers, and professional growth among other things.

If there is one book I can recommend anyone serious about developing their abilities and career and that of their teams, it is this book. Be forewarned the some of the ideas challenge conventional wisdom we have all come to accept, and might lead you with what I like to call '“brain-breaking moments”

In an era where your favorite AI tool can write complex code, the conventional wisdom of becoming a narrow specialist starts to seem increasingly precarious. The tech industry’s long-standing mantra of “go deep, not broad” is being challenged not just by AI’s capabilities, but by the very nature of innovation and problem-solving in our complex world. Don’t believe me? Try vibe-coding and see how quickly you can get your first MVP off the ground.

Epstein’s work shatters the popular “10,000-hour rule” now entrenched in popular contemporary professional development literature. It challenges our deeply held beliefs about early specialization and expertise. Through compelling research and fascinating stories - from Roger Federer’s late specialization in tennis to Nintendo’s journey from a playing card company to a gaming giant - the book reveals how individuals with broad experiences and diverse skill sets often outperform deep specialists, especially in fields characterized by rapid change and uncertainty. Not quite what we have known to believe.

For tech professionals, this message couldn’t be more timely. As AI continues to automate routine technical tasks and reshape traditional roles, the ability to adapt, connect disparate ideas, and see the bigger picture becomes increasingly valuable. The future belongs not to those who can write the most efficient algorithm, but to those who can understand the human context, spot unconventional opportunities, and navigate complex systems thinking.

What makes “Range” particularly relevant for today’s tech professionals is its challenge to the traditional career ladder. In an industry where the pressure to specialize early and deeply is intense, Epstein’s research shows that late specialization and diverse experiences can lead to more innovative thinking and better problem-solving abilities. This isn’t just about career survival in the AI era; it’s about thriving through intellectual agility and adaptability.

As we dive deeper into “Range,” I’ll demonstrate how its principles can be applied specifically to tech careers in an AI-transformed world. We’ll examine why some of the most successful tech leaders aren’t pure technologists but rather individuals who’ve cultivated broad perspectives and diverse skill sets. More importantly, we’ll discuss practical strategies for expanding your own range while leveraging your existing technical expertise.

Whether you’re a data scientist feeling boxed in by your specialization, a software engineer wondering about your role in an AI-driven future, or a tech leader looking to build more adaptable teams, this journey through “Range” will challenge your assumptions and open new possibilities for growth.

Get ready for some (again) brain-breaking moments as we challenge conventional wisdom and explore why range isn’t just a nice-to-have anymore - it’s the new black in professional success. You don’t want to miss this series. Go ahead and subscribe. More coming soon.

Subscribe now

Until next time,

Vijay

PS. If you missed the cultural reference to the title. See OITNB here: Orange is the new black

PPS. If you really liked this, please consider sharing with a friend you think might find this of value. I would really appreciate that!

Today, I want to make a case why influencing might be one of the most valued but least understood and appreciated skills among many of the brilliant tech professionals, including those in my own field of data science and more broadly data & analytics. Every aspect of what I share below broadly also applies to technology oriented fields.

There’s a quiet frustration that ripples through many senior technical professionals — engineers, data scientists, architects, and technical leads who have mastered the craft of precision but find themselves unheard in the rooms where direction is set. How do I know it? I have personally lived it!

Their ideas are solid, their logical considerations airtight, yet their influence seems to fade when conversations shift from design artifacts to business outcomes.

It’s not because they lack ability. In fact, they often hold more technical clarity than anyone else in the room. But their brilliance can feel wasted on executives who don’t fully grasp algorithmic sophistication or appreciate the intricacies of for example a topic on “system design”.

The problem isn’t their expertise — it’s the translation gap between technical depth and strategic understanding. It’s the old adage. They’re speaking “Latin” to an audience that needs “English.”

This article explores why that gap exists, why it persists even among the most gifted professionals, and what it takes to evolve from a technical authority into a strategic influencer without diluting technical integrity.

In the sections follow I will introduce the multiple dimensions that explains the gap as well as what I think are the 8 mental architectures understanding which can help forge a path forward.

The Hidden Paradox of Technical Mastery

For many senior technical professionals, excellence has always been a matter of precision including getting the details right, identifying edge cases others overlook, building solutions that stand up to complexity, and work under any circumstances.

That same instinct for perfection, however, often becomes the invisible barrier to influence.

Early in a career, mastery earns credibility. The person who can debug the impossible issue, optimize the slowest query, or architect a highly performant and scalable system becomes indispensable. But as responsibility scales, influence no longer depends solely on technical correctness, it hinges on strategic clarity.

The irony is that the very habits that fuel technical mastery including deep focus, analytical rigor, a love for complexity can make it harder to connect with audiences who value brevity, clarity, and decision speed.

Executives, after all, don’t reward intellectual elegance; they reward business alignment.

To them, the conversation isn’t about architecture patterns — it’s about risk, timing, and impact.

The engineer who wants to discuss trade-offs in depth might unintentionally appear indecisive to a leader who simply needs to know whether the project will ship on time.

This isn’t a failure of intelligence. It’s a mismatch of context.

Technical professionals are trained to optimize for truth and precision. Executives optimize for clarity and direction.

When these value systems collide, even the most brilliant individual contributors can feel sidelined. Not because they’re wrong, but because they’re not being heard.

The Mental Architecture of the Brilliant but Stuck

When analyzing why so many exceptional technical minds struggle to expand their influence, a pattern begins to emerge: a shared “mental architecture” shaped by habits that once drove success but now limit visibility.

Below are eight recurring dimensions of that architecture. None of them reflect a lack of intelligence or ambition; instead, they reveal how excellence in one domain can unintentionally create blind spots in another.

1. The Pursuit of Perfection

Perfectionism often disguises itself as professionalism.

For the technical expert, “good enough” can feel like moral compromise. They equate high standards with identity. Because, precision is who they are.

But in strategic environments, perfectionism often manifests as paralysis. Projects stall while the technically-minded person refines, polishes, double-checks, and triple-checks.

Meanwhile, those who can tolerate ambiguity move ideas forward and are perceived as more effective.

The challenge isn’t about lowering standards; it’s about reframing excellence from flawlessness to effectiveness.

2. The Executive Translation Gap

Ask a technical leader a question like, “When will this be ready?” and the instinctive response might begin with context: the architecture, dependencies, possible bottlenecks.

By the time they reach the actual answer, the executive has mentally moved on.

This isn’t a communication failure, it’s a framing mismatch.

Executives operate in compressed timeframes. They want the answer first, the rationale later.

Those who can distill complex work into clear, outcome-oriented language are not oversimplifying — they’re building bridges of understanding.

3.Blind Spots in Organizational Dynamics

Many technically brilliant people assume organizations operate as meritocracies that excellence speaks for itself. Most organizations are. But in reality, there is a less understood dynamic. It is flow of influence through human networks, not just performance metrics.

Ignoring how decisions actually move through an organization such as who influences whom, how priorities get shaped often leads to brilliant ideas dying quietly in the middle layers.

Strategic influence requires not only insight but navigation: understanding alliances, timing, and informal decision channels that rarely appear on an org chart.

4. Reluctance to Take Initiative

In highly technical environments, structure and clarity are virtues. Requirements, specifications, and defined roles ensure stability.

But at senior levels, waiting for permission before acting can be mistaken for hesitation. Some of the most capable engineers hold back brilliant ideas until they’re formally invited to lead them. They want certainty, clear scope, and alignment before moving.

Yet influence often grows from creating clarity, not waiting for it.

Strategic leaders shape direction by defining opportunities others haven’t yet articulated.

They don’t discard rigor; they pair it with courage to move even when the path isn’t fully marked.

5. Limited Cross-Functional Reach

Many technical professionals feel most at home among peers who share their vocabulary and logic. Conversations outside the technical circle can feel inefficient or superficial.

However, leadership visibility expands through connection, not just competence.

When relationships remain confined to technical peers, the person’s impact remains similarly contained.

Strategic influence, by contrast, requires relationships with business counterparts such as operations, marketing, finance, and product — not for the sake of checking the box on ‘are you being collaborative?’, but to understand the business holistically.

Those who cultivate cross-functional rapport start hearing early signals about market shifts, customer needs, and organizational priorities.

They stop being seen as “the technical expert in the corner” and start being viewed as “the person who understands how technology drives the business.”

6. Invisible Excellence

Another subtle barrier: the assumption that great work naturally gets noticed.

Many technical minds view packaging, presentation, or storytelling as distractions from “real” work.

Yet the corporate world runs on what I like to call narrative visibility.

When achievements are not communicated in a way executives can grasp, they remain invisible, no matter how impactful they are in code or data.

Framing technical outcomes in terms of customer impact, cost savings, or risk mitigation isn’t embellishment — it’s translation.

It ensures that the organization recognizes the strategic value embedded in technical accomplishments.

7. Disconnection from Market Context

It’s easy to get absorbed in optimizing internal systems and technology deliverables, but without a clear sense of the market landscape — competitors, customers, revenue models — technical decisions risk becoming detached from strategic priorities.

Understanding business strategy isn’t about abandoning engineering purity; it’s about designing with contextual intelligence.

When technical professionals understand where the company is trying to go, they can proactively align architectures, data models, or algorithms with that trajectory.

Strategic influence grows when technical decisions reflect awareness of business constraints and opportunities.

8. Mixed Leadership Signals

Even at senior levels, many technical experts unconsciously send signals that anchor them as deep specialists rather than organizational leaders.

They dive into technical minutiae during executive discussions, optimize for precision over persuasion, and often speak in frameworks rather than outcomes.

Strategic leaders, by contrast, send a different signal: confidence in ambiguity, focus on direction, and fluency in organizational language.

This doesn’t mean abandoning one’s craft — it means widening the aperture through which expertise is expressed.

Influence is as much about how thinking is perceived as it is about what thinking produces.

Why Executives See the World Differently

Executives operate in a different mental environment. Their bandwidth is stretched across dozens of priorities — customers, investors, markets, competitors, talent, and timelines.

While technical professionals often look inward (to systems, processes, and precision), executives look outward — to risk, reputation, and results.

They interpret information through the lens of decision velocity: What does this mean for our objectives, and what should we do next?

This perspective isn’t better or worse — just different.

Technical experts who understand this distinction begin to see that communication with executives isn’t about simplifying ideas; it’s about framing insights within the decision horizon of leadership.

For example, when discussing a new data-infrastructure upgrade, the executive mind seeks to know:

• What business pain does this solve?

• What risk does it reduce or opportunity does it enable?

• How does it align with our timing and budget reality?

Once those anchors are clear, details find their place naturally.

Strategic influence begins when the conversation shifts from how the system works to why the system matters.

The Cost of Staying in Translation Mode

Remaining trapped between technical depth and organizational influence carries emotional and career costs.

Many senior engineers and data professionals experience a quiet sense of stagnation — the feeling of being indispensable yet peripheral.

Their work powers key decisions, yet they’re rarely part of the decision-making itself.

They see less technically skilled peers shaping direction simply because those peers can speak the language of outcomes.

Over time, this creates fatigue — a subtle disillusionment that technical merit alone should be enough.

But influence is not a reward for brilliance; it’s a by-product of relevance.

When ideas don’t travel upward, potential impact stays localized.

Bridging that gap requires intentional evolution — not in skillset, but in mindset.

Bridging the Divide: From Expert to Strategic Partner

Bridging the space between technical depth and organizational strategy is less about abandoning expertise and more about expanding perspective.

It’s a deliberate evolution — from contributor to translator, from problem-solver to sense-maker.

This transition usually begins with three intertwined shifts.

1. From Precision to Perception

Precision builds systems; perception builds alignment.

The technically minded professional knows how things should work.

The strategically minded leader also senses what the organization is ready to hear.

Strategic perception means reading the room — noticing who needs confidence, who needs clarity, and when to stop explaining because the decision is already emotionally made.

Influence grows not from winning every argument but from understanding which ones actually shape momentum.

2. From Output to Outcome

At advanced stages of technical careers, success is no longer measured by how much is built but by what it enables.

The most respected strategic leaders speak the language of outcomes: customer retention, revenue protection, cost avoidance, market resilience.

They reframe metrics from “system uptime” to “business continuity,” from “algorithm accuracy” to “decision quality.”

When technical work is expressed through outcomes, it gains executive gravity.

3. From Isolation to Integration

Strategic influence comes from building connective tissue across the enterprise.

When technologists learn to integrate their work with marketing narratives, finance models, or customer journeys, they become translators of value.

Their ideas stop being “engineering initiatives” and start being “organizational levers.”

The result isn’t dilution — it’s amplification.

The same technical brilliance, viewed through a strategic lens, becomes exponentially more influential.

Building Influence Without Losing Technical Integrity

A common fear among deep technical professionals is that becoming more strategic will somehow make them less technical — that communicating simply or talking about business value equates to “dumbing down.”

In reality, it’s the opposite.

Strategic communicators elevate their technical work by revealing its meaning to those who fund, prioritize, and scale it.

They don’t abandon depth; they decide how much of that depth each audience actually needs.

Some practical ways this shows up:

• Tell the story of impact. Replace data dumps with narratives: the problem, the tension, the change achieved.

• Use visuals for clarity, not complexity. Architecture diagrams should guide decision-making, not prove sophistication.

• Practice executive brevity. Start with the conclusion; then, if invited, unpack the reasoning.

• Translate metrics into business equivalents. Latency becomes customer experience; data-pipeline reliability becomes revenue assurance.

• Share ownership of ideas. Invite non-technical colleagues into design conversations early; co-creation builds advocacy.

These habits turn technical precision into organizational language.

Over time, the person who once “built the system” becomes the person trusted to shape where the system should go next.

The Path Forward

The gap between technical brilliance and strategic influence isn’t a flaw in ability — it’s a predictable stage in professional evolution.

Most organizations unintentionally create it by rewarding depth early and expecting breadth later, often without guidance on how to cross the bridge.

Those who do make the transition discover something surprising: they don’t lose their edge; they gain range.

They stop being the smartest person in the meeting and start being the one who can make everyone else smarter.

They move from being experts in technology to becoming architects of understanding — individuals who can translate complexity into clarity, and clarity into action.

The world needs more of these translators.

Because the future of technology-driven organizations depends not only on how well systems perform but on how effectively people align around their potential.

A Gentle Invitation

If this reflection resonates and if you recognize parts of yourself in these patterns and are curious about what deliberate evolution might look like, consider subscribing to my writing by clicking the button below.

That’s where I write more deeply about the craft of transforming technical excellence into strategic impact, about how influence really grows inside organizations, and about the quiet mindset shifts that turn brilliant experts into trusted partners in shaping direction.

I also occasionally email useful information to my subscribers not shared elsewhere.

Your technical mastery doesn’t need to shrink for your influence to grow. It just needs to speak a language the organization can hear.

Until next time,

Vijay

PS. If you really liked this, please consider sharing with a friend you think might find this of value. I would really appreciate that!

Share The Fractal

Remember when "I Googled it" became our universal response to any question? Now, with ChatGPT and other Large Language Models (LLMs) in a close horse-race to become the Master who 'knows it all' , that phrase seems to be slowly evaporating from our vocabulary – like water droplets imperceptibly leaving an ocean. While Google isn't disappearing anytime soon, our approach to information seeking is undergoing a fascinating transformation.

Ironically, as research for this post, I took the traditional route. I googled "data skills" to see what the well-SEO'd, "authoritative" sources would reveal about the essential skills for data science. Surprisingly, across several pages of results, I couldn't find a single mention of what I consider the most crucial skill:

Analytical Thinking.

You might ask, "What exactly is analytical thinking?" Let me explain: Analytical thinking is a multi-dimensional convergence of skills that combines different competencies to optimize (mathematically speaking) the most efficacious answer to the most consequential problem.

While this might sound obvious, there are several non-obvious challenges in data science (more broadly - pattern recognition; data science is inherently pattern recognition) where analytical thinking proves essential:

The missing "Meta" of the Problem

In business settings, we often tackle problems that are either nuanced or remarkably different from what we should be addressing. Why? Because we're missing the crucial context, origin, and ethos surrounding the problem. Each of these elements is vital (I'll delve deeper into these aspects in future posts).

The Toolbox "Friction"

Many of us develop a bias toward tools we've mastered, creating resistance to learning new approaches that might offer different perspectives. What we need is a heuristic map – similar to how emergency rooms have standard operating procedures for various scenarios. While medicine is a science, the ER's intervention process is a combination of science and art. Science as in algorithms making data driven decisions in critical cases such as trauma, cardic arrest, stroke. Art as in human judgment using their experience and intuition to apply pattern recognition and making fast decisions in the most ambiguous situations. ERs don't have a toolbox friction, the cost of that friction is too high. The cost of that friction is a human life. I would argue, these ER doctors would be the savviest analytical thinkers through their gruelling training. Most other places, we are afforded to carry this friction with us, as the cost equation is entirely different, or perceived to be non-consequential, which unfortunately is not true.

Bringing Guns to a Stick Fight

Sometimes simple is not only better. Simple is the only viable answer. Just because a complex answer can be built, doesn't mean it is the right answer. I mentioned ethos earlier. Depending on the industry, domain and yes, even organization, data-centricity culture plays a huge role in how we go about producing meaningful data-driven capabilities. This is compounded by what I call a 'Desire for complexity' bias. A lot of us data science professionals go through reasonably complex graduate math materials. There is a need to resist the urge to wield the laser gun in a stick fight.

There are more of these types of issues require the development of Analytical Thinking skills. And that is fundamental to success in data science and data analytics careers.

If you're:

• Aspiring to build a career in Artificial Intelligence and Machine Learning

• Curious about applying pattern recognition skills in various areas of life, or simply want to stoke your curiosity.

Follow me here

Substack, LinkedIn, Twitter

I'll be sharing detailed insights on developing these crucial analytical thinking skills in future posts.

This article describes how amazon reviews can be interpreted and classified back into a low or high rank using a bag of words vectorization approach to generate sentiment based on the comment or review left by the buyer.

We perform analysis of comments submitted by buyers of a specific product on Amazon.com and applying NLP techniques to the reviews combined with overall score (1 to 5) as provided by the buyer.

For the purpose of this analysis, a single product referenced by ASIN (Amazon Standard Identification Number), akin to SKU on a retail outlet floor was used. This product name as sold on Amazon is “AcuRite 00613 Digital Hygrometer & Indoor Thermometer Pre-Calibrated Humidity Gauge”.

At the time of this analysis, it retailed for less than $15, under product category of “Industrial & Scientific”. And, there were 21K ratings viewable on amazon.com. The dataset that is used from an archive obtained for this project has 1,229 reviews for this product specifically, along with the detailed text review as well as product rating.

The objective of this analyis is to predict what rating might the user provide based on the sentiment expressed in the comments. As Amazon ratings are on a 1 to 5 scale, for sake of simplicity, this 5-class problem was created as a 2-class problem by combining ratings {1,2,3} into “Low” rating, and, ratings {4,5} into “High” rating.

How natural language processing is applied in this analysis:

There are three main steps in this analysis.

(1) Data acquisition and pre-processing of the text data.

(2) Vectorizing the data using bag of words.

(3) Applying Naïve Bayes’ classifier on this vectorized text data to create a predictive model that takes a comment and predicts the user sentiment as Low or High.

A preview of model results

The classifier is highly accurate in predicting both “High” and “Low” ratings, with a F1-score of 0.96 & 0.97 for each class respectively. On inspecting the actual comments, we can see that the predictor did a good job of attributing sentiments such as “works really good”, “works as advertised”, “awesome addition” to a High rating, and attributing sentiments such as “why bother with reviews”, “way off”, “worked fine for a month, but” to a Low rating. Though you will see the way the model in this analysis actually works is not by inferring the sentiment based on sequence of words as described above in quotes, but rather on the count of words. M

We close this analysis with some limitations and improvements (and if you guessed Transformers as options — you are right!). So, let’s review:

Step 1: Data Acquisition and Preparation

The data used for this analysis is different from a standard “corpora” dataset in that the original data owned and tracked by Amazon.com is stored as a tabular data with 12 attributes. Of interest to this analysis are only few attributes that were retained. It includes the following:

· reviewText — The actual text comments entered by a buyer

· summaryStr — Summary of the comments as provied by the buyer

· overall — The original rating on a 1–5 scale.

· ASIN — This is the unique ID for a product as noted earlier.

The data used for this analysis was obtained from an archive on ucsd.edu domain. The original file is in JSON format and has 77,071 reviews.

As the original file obtained for the analysis had ratings and comments for a number of products under the category of “Industrial & Scientific”. We picked one specific ASIN (product) on which this analysis could be performed. A group by operation was performed using valuecounts() function to get count of records based on ASIN, and the one with the most records were used for this analysis. The rationale for choosing ASIN with highest number of records was so that we could work with a reasonably larger number of records to train and test the classifier. The finally processed dataset was of the shape 1229 x 4. The 4 members in the DataFrame are of “object” type, and therefore need to be typecasted into string (for review) and int (for rating) formats.

from urllib.request import urlopen
from nltk.corpus import stopwords
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import classification_report
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.metrics import confusion_matrix
import nltk
import os
import json
import gzip
import pandas as pd
import wget
import matplotlib.pyplot as plt
import seaborn as sns
import string

url="http://deepyeti.ucsd.edu/jianmo/amazon/categoryFilesSmall/Industrial_and_Scientific_5.json.gz"
textfile=wget.download(url)

data = []
with gzip.open(textfile) as f:
    for l in f:
        data.append(json.loads(l.strip()))
    
# total length of list, this number equals total number of products
print(len(data))

# first row of the list
print(data[0])


#quick review of data structures - list, df
df = pd.DataFrame.from_dict(data) #convert dictionary  to dataframe

#check length of dataframe
print(len(df))

#check content of first record
df.iloc[0]
      
#check comments of first record
df.reviewText.iloc[0]

#check summary stats of the dataframe
df.asin.describe()

#alternative way to find the most frequently occuring ASIN to identify candidate product for analysis
df['asin'].value_counts() #shows B0013BKDO8  has the most ratings

asindf=df[df['asin']=='B0013BKDO8'] # keep product of interest

#check shape of new df
asindf.shape #reduced to a single ASIN as above.

asindf=asindf[['asin','overall','reviewText','summary']] # keep four cols

Basic Exploratory Data Analysis

Here is a quick view of what this datset looks like. The last column called newrating was created by additional data processing described below.

There are only four variables originally retained, so EDA is quite quick to do. First, let’s see textLength.

The above chart shows the distribution of the length of the text comments buyers provide in addition to their ratings. And this is what the distribution looks like. Average length of the string is 153 chars with a standard deviation of 239. This is a highly skewed distribution.

import klib as kl
kl.dist_plot(finaldf.textLength)

Interestingly, longer text is associated with higher ratings as seen in this box plot.

plt.figure(figsize=(15, 5))
sns.boxplot(data=finaldf, x="rating", y="textLength")

Next, we’ll see the rating.

This is an interesting observation, ratings are heavily skewed towards 5, then followed by 4. This must be one of those really well performing products for most buyers. I certainly can vouch for that that because I own one of this, and it is always by my bedstand. Here is the picture of the device I bought many years ago (still working well!)

The correlation of text length to rating is very low (0.18). As we can also see beow there is hardly anything discernable about the impact of comment length to the 5 separate ratings.

We binarize the ratings with a new attribute called newrating, which has two values {Low, High}. Ratings 1,2,3 are binned under Low. Ratings 4,5 are binned under High.

# derive new rating
def set_new_rating (row):
   if row['rating'] == 1 :
      return 'Low'
   if row['rating'] == 2 :
      return 'Low'
   if row['rating'] ==3 :
      return 'Low'
   if row['rating'] ==4 :
      return 'High'
   return 'High'


finaldf['newrating'] = finaldf.apply(set_new_rating, axis=1)

Step 2: Text Pre-processing for natural language processing

There are a few key elements that are being used for text preprocessing. We do the following steps:

a. removing punctuations by reading comments.

b. removing English stopwords.

c. returning a list of clean tokenized words.

The reviewText before and after this processing appears as shown below:

Following code block shows the text pre-processing:

from nltk.tokenize import word_tokenize


def text_processing(varText):
    review = [char for char in varText if char not in string.punctuation]
    review= ''.join(review)
    return [word for word in review.split() if word.lower() not in stopwords.words('english')]

For removing punctuation, each character is checked against the String.punctuation — a constant string that is a property of the string object with a value of ‘’!”#$%&\’()*+,-./:;<=>?@[\]^_`{|}~’
Where a match isn’t found, that character is retained and joined back into a string. Therefore, this string was split again to create words, to then compare against English stopwords imported from NLTK.corpus. Examples of stop words include words that are frequently used in English which typically don’t carry weight in most text analysis such as “I”, “me”, “myself”, “we”, etc. Each reviewText was processed to generate tokens in this manner

We could have also applied stemming and lemmatization (tokenizing similar words down to a single word e.g. did, doing, done are treated as the same word). This allows similar words to be tagged as the same word allowing for a more precise weight calculation of the presence of such words. It was left out in this exercise, however.

Vectorization

Each review is converted as a list of tokens. Each review is now converted as a vector to train the classifier model. We apply bag of words (bow) model to this text. We use Scikit Learn’s CountVectorizer to convert word in each of the comments into a matrix of token counts (not the token itself — since, we already achieved that in the previous step). This allows creating a vocabulary of 2,326 words in the dataset and counts occurrence of each word. This same step is applied to individual comments to create a matrix of 1,229 records X 2,326 words. We get a sparse matrix of this dimension with a sparsity index of 0.57%.

An individual record is profiled here for illustration. The 8th reviewers’s comment in the dataset is as follows.

import textwrap
reviewer8 = finaldf['reviewTextStr'][7]
print('\n'.join(textwrap.TextWrapper(50).wrap(reviewer8)))

I love this little unit. It helps me when I am
feeling cold to re-assure me that I indeed don't
need to cut on the heater, and that I just need to
wait for my body heat to normalize to the
environment. Also, it has helped when I
accidentally leave the heater on during a nap. I
can say that, thanks to this unit's Low Temp /
High Temp recording feature, I was able to find
out that I had left the heater on by mistake and
achieved a whopping 95 degrees (F) out of the
space heater.

The count vectorization code follows below along with a vector representation (truncated for brevity) of reviewer8.

bow_transformer = CountVectorizer(analyzer=text_processing).fit(finaldf['reviewTextStr'])
# Print total number of vocabulary words
print (len(bow_transformer.vocabulary_))
bow_transformer.vocabulary_ 

The vocabulary is the entire list of unique words found in the preprocessed text by the number of occurences. It is easy to see (but not always true) what words might accompany a higher ranking vs a low ranking. Some examples are highlighted in red. Also, this is a single token (single word) count. We are not looking at a n-gram (n word sequence) to evaluate the sentiment. For instance, we notice the word below, “happy”, it is currently taken in isolation. The sentiment can easily point in the negative direction and lower the ranking we are trying to derive, if “happy”is preceded by the word “not”. So that is definitely an improvement that can be made using a bi-gram or n-gram tokenization.

bow8 = bow_transformer.transform([reviewer8])
print(bow8)

This produces forty tokens from the original comment found in reviewer8 record. Some examples are shown below.

#check  the English word for the feature in below vectors

print (bow_transformer.get_feature_names_out()[386]) #Temp
print (bow_transformer.get_feature_names_out()[1153]) #heater

Now that we have generated the vocabulary, we need to establish the entire corpus of preprocessed reviews as a matrix of tokens. Naturally, we expect the number of rows in the matrix to be the same as number of reviews, and the number of columns in the matrix to be the same as the length of vocabulary. This brings us to the shape 1,229 records X 2,326 words, we already established above.

This is the code to generate the matrix of count vectors.

comments_bow = bow_transformer.transform(finaldf['reviewTextStr'])
print('Shape of Sparse Matrix: ', comments_bow.shape)
print('Amount of Non-Zero occurences: ', comments_bow.nnz)
print('sparsity: %.2f%%' % (100.0 * comments_bow.nnz / (comments_bow.shape[0] * comments_bow.shape[1])))

Now that we have laid the numeric ground work much needed for computation, we still have to arrive at a metric that tells us importance of a word relative to another. That is precisely what TFIDF does. TFIDF stands for Term frequency inverse document frequency. It is the product of the two terms — Term frequency and Inverse Document Frequency. Term frequency is the ratio of number of times a word occurs in a document (in our case, the entire set of reviews used for this analysis) over the total number of words. So higher the ratio, the more important the term is. Whereas, Inverse Document Frequency is the log of the ratio of total number of documents over number of documents where the term is found. Where does the inverse word come from in IDF? It comes from the idea that this term can also be expressed as the log of the inverse of the ratio of documents where the term is found. It semantically has the same meaning as our first formal definition of IDF. Basically, it’s like saying x is also the inverse of 1/x.

We check the TFIDF values of some words we think are important.

tfidf_transformer = TfidfTransformer().fit(comments_bow)

print(tfidf_transformer.idf_[bow_transformer.vocabulary_['recommend']])
print(tfidf_transformer.idf_[bow_transformer.vocabulary_['great']])
print(tfidf_transformer.idf_[bow_transformer.vocabulary_['fantastic']])
print(tfidf_transformer.idf_[bow_transformer.vocabulary_['amazing']])
print(tfidf_transformer.idf_[bow_transformer.vocabulary_['love']])
print(tfidf_transformer.idf_[bow_transformer.vocabulary_['bad']])
comments_tfidf = tfidf_transformer.transform(comments_bow)

Recall, our goal was to take all the text ratings and see if we can classify them into a high or low ranking. So, for that we check the distribution of our derived rating from the newrating field.

newaxisvals = finaldf['newrating'].value_counts()
plt.figure(figsize=(30, 10))
plt.subplot(131)
plt.bar(newaxisvals.index,newaxisvals.values)
plt.xticks(fontsize=18)
plt.yticks(fontsize=18)
plt.show()

This is an unbalanced distribution, which we would have expected based on the distribution of the original ratings we saw in our initial analysis.

distratio = finaldf['newrating'].value_counts()[0] /  finaldf['newrating'].value_counts()[1] 
print(distratio) # gives a value of 6.68125

The number of High ranking records is 6x times that of Low ranking. In other words, the ratio of majority class over minority class is 6x. So, before we train a classifier, we need to balance this data. Otherwise, the classifier will have a bias to incorrectly predict ratings with High ranking more than it would for Low ranking.

We apply random oversampling where the minority class is resampled so that the High and Low rankings are equally distributed. We use RandomOverSampler class to do this.

from imblearn.over_sampling import RandomOverSampler
oversampler = RandomOverSampler(sampling_strategy='minority', random_state=42)
newx, newy = oversampler.fit_resample(comments_tfidf,finaldf['newrating'])
#when you print newx you will see a sparse matrix of float64 dtype with 2138
#obsevations (upsampled from previously 1229 observations)

And a quick check shows that the data is now balanced.

newy.value_counts()
newaxisvals = newy.value_counts()
plt.figure(figsize=(30, 10))
plt.subplot(141)
plt.bar(newaxisvals.index,newaxisvals.values)
plt.xticks(fontsize=18)
plt.yticks(fontsize=18)
plt.show() 

Now that we have the data ready for training the model, we partition it into test and train and then train the model, followed by checking how we fare on predicting by comparing the predicted ranking with actual ranking.

MultinomialNaiveBayesClassifier is a good model for this work as it takes a sparse matrix as an argument as X, which is exactly what we have setup and takes an array for Y. In our case it is the sentiment (ranking).

comment_train, comment_test, rating_train, rating_test = train_test_split(newx, newy, test_size=0.3)
rating_detection = MultinomialNB().fit(comment_train, rating_train)
rating_pred = rating_detection.predict(comment_test)
print(classification_report(rating_test, rating_pred))

We see a pretty good F1 score, which is a harmonic measure of precision and recall combined together as shown in the classification report below.

Notice both High Ranking and Low Ranking are predicted equally well.

It may be worth reiterating the significance of balancing the data. If we don’t balance the classes, the F1 score for Low Ranking suffers giving us a value of 0.5. If you are interested in seeing if this is true, you might want to run this experiment without the oversampling. Without balancing (6:1 ratio of High:Low ranking in this case), this bias skews the overall accuracy to 0.91 showing the model is highly accurate when in fact it is not.

#print confusion matrix
cmat = confusion_matrix(rating_test, rating_pred)
plt.figure(figsize=(10,10))
sns.set(font_scale=1.5)
sns.heatmap(cmat.T, square=True, annot=True, fmt='d', cbar=False,
            xticklabels=['High','Low'], yticklabels=['High','Low'])

plt.xlabel('True Rating')
plt.ylabel('Predicted Rating')

The confusion matrix below shows the true positives, true negatives, false positives and false negatives.

Limitations & Improvements

There are many improvements that can be done with this experiment. We used word count as the vector for the text representation. This experiment was done for a specific type of niche product among the millions of products sold on Amazon. This specific method of vectorizing using bag of word (BOW) technique introduces sparsity exponentially as more vocabulary is introduced. Which means if there were a lot more reviews, or we open this analysis to a broader array of products, we’d run into computational issues due to the sparsity. BOW also does not take into account word order and meaning of the word. While this is a simpler technique, it has been extensively used in the past for document and text classification.

There are better ways to handle text classification. Word embeddings offer a denser representation than what we saw. It is therefore computationally efficient. By combining word embeddings with neural networks different bodies of text such as documents (or in our case reviews) can be classified.

Lastly, I’d be remiss to say that Transformers is yet another and newer way to handle text classification. I’ll cover these in future articles in natural language processing using transformer architectures.

Until next time!

References:

Amazon review data

An approach to applying log linear models (Generalized Linear Models) on categorical data with applications in ecological studies.

As data science practitioners, we arm ourselves with standard tools in our belt which we tend to put to use when we frame the problem domain to be one of classification or regression problem(for supervised learning), or a clustering problem (for unsupervised learning). These tools give a pretty good mileage for getting results on implementing machine learning solutions when applied to structured data.

In this article, I discuss a robust modeling tool to work with categorical data: Generalized Linear Model (GLM).

GLMs are widely used in social sciences, market research, health sciences where categorical scales are predominantly used. In financial services, insurance industry also use GLMs extensively. In market research, for example, a customer preference study done by a market research company on their customer panel between choice of a brand for a shampoo might ask them to state their choice between those made by L’Oreal, P&G and Unilever. The modeling here is used for identifying how to explain a customer’s choice or preference based on other variables (aka covariates), each of which is represented further in categorical scales.

What is categorical data?

Consider the following examples —

Ordered categorical data: A customer service survey showing customer respondes rating the service across choices {poor, ok, excellent}. There is a definite order from low to high in this list of categories.

Unordered categorical data: In a consumer preference study, a demographic variable of where the consumer lives could be a variable. Options may include {urban, suburban, rural.. etc}. There is no concept of order for this option.

GLMs are great tools to work with when working with studies that require analysis and modeling based on this type of data.

What is a Generalized Linear Model?

Before we get into the how, let’s briefly look at what GLMs are.

GLMs are a broad set of models that includes linear regression and ANOVA models for continuous data as response variables, as well as models for discrete responses. One such specialized cases of a discrete response model, for binary response is logistic regression model. This modeling technique is used for credit and other risk modeling in financial services as its known for its effectiveness, simplicity and parsimony as it applies to this domain. GLMs also allow for modeling a count (frequency) based response variable.

A GLM is made of three components —

1. A random component (Y) wherein we assume there is a certain probabilistic distribution.

2. A systematic component which represents the covariates or explanatory variables for the model.

3. A link function which represents the function of the expected value of Y. And this function allows Y to be expressed as some linear combination of the explanatory variables.

Modeling and Analysis with GLM

Next, let’s look at the analysis and modeling using GLM, along with code.

In this analysis we use a small dataset from the field of ecology where the day time habitat information of two lizard species — Grahami and Opalinus were collected. This data was obtained by observing specific information about habitat locations and the time when it was observed (McCullagh and Nelder, 1989).

Let’s look at the data to make sure we understand what an entire dataset of cateogorical data type looks like.

Lizard Species (L): Grahami or Opalinus indicating which species was observed. This is our target variable.

Illumination (I): indicating whether it was Sun or Shade.

Perch diameter (D): in inches with following values (aka factors): D≤ 2 and D> 2.

Perch height(H): in inches (H) with following values: H<5 and H≥ 5.

Time of day (T): also factor variable with three levels: Early, Mid-day and Late.

As we can see there are no continuous data, all of them are categorical data.

This modeling was implemented in R. You can implement this in your favorite language including Python. The concept of how to apply categorical data analysis and modeling is the same.

Let’s look at the data:

> lizard.data <-read.table(file = "lizard_data.txt",header = TRUE)
> head(lizard.data)
  Illumination Diameter Height   Time  Species Count
1          Sun      <=2    <=5  Early  Grahami    20
2          Sun      <=2    <=5  Early Opalinus     2
3          Sun      <=2    <=5 Midday  Grahami     8
4          Sun      <=2    <=5 Midday Opalinus     1
5          Sun      <=2    <=5   Late  Grahami     4
6          Sun      <=2    <=5   Late Opalinus     4

The data is setup in the form of what is called a higher order contingency table. We apply different log linear models on this data. We calculate the effect size, perform best model selection based on AIC. We also perform goodness of fit tests and residual analysis to assess model adequacy.

The steps taken for this analysis, output and discussion on the model performance are detailed below.

Fit log-linear models

Using the lizard data, we first fit three log-linear models including:

(i) Model of mutual independence,

(ii) Model of homogeneous association,

(iii) Model containing all the three-factor interaction terms.

# Create the features from the dataset
I<-factor(lizard.data[,1])
D<-factor(lizard.data[,2])          
H<-factor(lizard.data[,3])
T<-factor(lizard.data[,4])
L<-factor(lizard.data[,5])
Count<-factor(lizard.data[,6])

library(MASS) #need this package for using loglm 

##create the three models first

#create saturated model
liz.model.sat <- glm(Count ~ I*D*H*T*L, family = poisson, data=lizard.data)
liz.model.sat$formula

#mutual independence
liz.model.mi <- glm(Count ~I+D+H+T+L, family = poisson, data=lizard.data)
summary(liz.model.mi)

#homogeneous association
liz.model.ha <-glm(Count ~ I*D+I*H+I*T+I*L+D*H+D*T+D*L+H*T+H*L+T*L,
                   family = poisson, data=lizard.data)
summary(liz.model.ha)

## then, get LRT stat (deviance, or, g-sq), aic, pearson x-sq, p-value for  model

#mutual independence:
deviance(liz.model.mi) #152.6136 
df.residual(liz.model.mi) # 41
AIC(liz.model.mi) #329.8624
mi.1 <- loglm(Count ~ I + D + H + T + L, data = lizard.data)
x2.mi<-mi.1$pearson
x2.mi  #pearson chi sq is 157.8766
anova(liz.model.mi, liz.model.sat, test = "Chisq")
#p-value is 9.186e-15 ***

#homogenous association
deviance(liz.model.ha)#5.05375 
df.residual(liz.model.ha) # 27
AIC(liz.model.ha) #230.3026
ha.1 <- loglm( Count ~ I * D + I * H + I * T + I * L + D * H + 
                 D * T + D * L + H * T + H * L + T * L, family = poisson, 
               data = lizard.data)
x2.ha<-ha.1$pearson
x2.ha  #pearson chi sq is 20.93927

We then calculate the goodness of fit using residual deviance, which is the same as the G squared statistic (Goodness of fit), shown in Table 1 below.

Using AIC to determine best model

In order to get the goodness of fit, we need a saturated model, a model with df = 0. Unlike (binomial) logistic regression, we cannot solely use AIC because of possible conditional association between two or more terms in a log linear model. We begin the assesment by looking at model selection statistics including goodness of fit (LRT statistic, or deviance), p-value, AIC, Pearson Chi-squared value, and residual degrees of freedom.

Adding more complexity to the modeling

We evaluate the three models created in the previous section that is, single-factor terms, two-factor terms, and, all three-factor terms, and additionally setup a four-factor term based model (code below). Looking at rows 1 to 4 (Table 2), the mutual independence model fits very poorly.

## create a three factor interaction model, and get diagnostics

liz.model.trif <-glm(Count ~ 
                       I*D*H+I*D*T+I*D*L+I*H*T+I*H*L+I*T*L+D*H*T+D*H*L+D*T*L+H*T*L,
                     family = poisson, data=lizard.data)
summary(liz.model.trif)
deviance(liz.model.trif)#13.23126
df.residual(liz.model.trif) # 11
AIC(liz.model.trif) #250.4801
trif.1 <- loglm( Count ~ I * D * H + I * D * T + I * D * L + I * 
                   H * T + I * H * L + I * T * L + D * H * T + D * H * L + D * 
                   T * L + H * T * L, family = poisson, 
                 data = lizard.data)
x2.trif<-trif.1$pearson
x2.trif  #pearson chi sq is 11.85212

anova(liz.model.trif, liz.model.sat, test = "Chisq")
#p-value is 0.2785

#create four factor interaction model
liz.model.fourf <- glm(Count ~ I*D*H*T+D*H*T*L,family = poisson, data=lizard.data)
# #Illumination*Species*Height*Time possibly has a zero in cross-class tab, so term removed
summary(liz.model.fourf)
deviance(liz.model.fourf) # 14.63944
df.residual(liz.model.fourf) # 12
AIC(liz.model.fourf) # 249.8883
fourf.1 <- loglm( Count ~ I * D * H * T + D * H * T * L, family = poisson, 
                  data = lizard.data)
x2.fourf<-fourf.1$pearson
x2.fourf  #pearson chi sq is NaN.
anova(liz.model.fourf, liz.model.sat, test = "Chisq")
#p-value is 0.2617

Pruning too much complexity: Using StepAIC to evaluate

We need a model that is more complex than the homogeneous association model, but simpler than the three-factor and four-factor interaction term models. In order to get that, we then used StepAIC function by starting with the four-factor to evaluate models with different AIC, We also did the same step with three-factor term model. We picked the models in both cases with lowest AIC in either of them, and then proceeded to remove the redundant terms in each (picking models shown in Rows 5 and 6 in Table 2).

Based on this observation, we select row 6 (D,T,IH,IT,DH,DT,DL,TL,IHL) as the best model.

## setup multiple models, evaluate each using AIC and goodness of fit 
#use stepAIC to manually select few models that 'appear' good for evaluation.

stepAIC(liz.model.mi) #329.9, residual dev = 152.6
stepAIC(liz.model.ha) #225.7, residual dev = 28.44, 
stepAIC(liz.model.trif)#AIC: 227.4, residual dev =26.1. 
stepAIC(liz.model.fourf)#AIC: 249.9, residual dev=14.64

##pick the lowest residual models and remove redundant params to fit few models
##and choose one final best model

#model 1
liz.model.besteval.1 <- glm(formula = Count ~ I +  I:H + D:H + I:T + D:T + 
                              + D:L +  T:L + I:H:L+ D:H:T:L,  family = poisson, data = lizard.data)
#I:D:H:T + (fails to converge...so, term removed)
aic1<-AIC(liz.model.besteval.1) 
aic1 #238.254
d1<-deviance(liz.model.besteval.1)  
d1 # 15.00517
df.residual(liz.model.besteval.1) #18

anova(liz.model.besteval.1, liz.model.sat, test = "Chisq") #0.6616
besteval11 <- loglm(Count ~ I  + I:H + D:H + I:T + D:T + 
                      I:L + D:L + H:L + T:L + I:H:L + D:H:T:L,  family = poisson, data = lizard.data)

x2.bestevall11 <-besteval11$pearson
x2.bestevall11 #12.58482

#model 2
liz.model.besteval.2 <- glm(formula = Count ~ D + T +  I:H + D:H + I:T + D:T + 
                              D:L + T:L + I:H:L,  family = poisson, data = lizard.data)
aic2<-AIC(liz.model.besteval.2) 
aic2 #227.3522
d2<-deviance(liz.model.besteval.2)  #26.1033
d2
df.residual(liz.model.besteval.2) #29

anova(liz.model.besteval.2, liz.model.sat, test = "Chisq")#p-value is 0.62
besteval21 <- loglm(Count ~  D +  T + I:H + D:H + I:T + D:T + 
                      + D:L + T:L + I:H:L,  family = poisson, data = lizard.data)

x2.bestevall21 <-besteval21$pearson
x2.bestevall21#25.28304

liz.best=liz.model.besteval.2 #best model.
deviance(liz.best)

Residual analysis of best model

Statistics for goodness of fit calculated above show a model that appears to fit to the data reasonably well. It is important that residuals are evaluated at each cell to show the quality of fit. Residual analysis can show that some cells may display lack of fit in a model that otherwise may appear to fit to the data well. We calculate standardized residual using rstandard function, where the observed and fitted counts are divided by their standard errors shown in Table 3 below.

Table 3 output was generated using the following code:

#get residuals for best model 
pred.best<-fitted(liz.best)
infl<-influence.measures(liz.best)  #residual diagnostics
influencePlot(liz.best) 
influenceIndexPlot(liz.best)
std.resid<-rstandard(liz.best, type = "pearson") # standardized resid
likeli.resid<-rstudent(liz.best, type = "pearson") #likelihood resid
liz.resid<-data.frame(I,D,H,T,L,Count, pred.best, std.resid, likeli.resid)
write.csv(file="lizard_residuals_pearson.csv", liz.resid)

We perform several checks on the model including influencePlot. Figure below shows a few observations as outliers. A studentized residual value greater than equal to 2 or less than or equal to -2 is considered an outlier. We see two cells, a total of 5 count of lizards as outlier. The size of the bubble indicates measure of the influence of the observation, (it is the square root of cook’s distance).

Framing this as a logistic regression problem

We can also do a logistic regression where it would require us to transform the 48 record dataset to a 564 record dataset where each record represents the Grahami or the Opalinus species. That would allow us to predict for the given I,D,H,T condtions what lizard species might we find. In order to pursue this method, we convert lizard as response variable.

Data was converted to a long table with 564 observations, and each variable was recoded 0,1 for all variables and 1,2,3 for T. The header records looks like this:

library(car) # for using Anova
lizard.longdata <-read.csv("lizard_final_binomial.csv")
lizard.longdata
head(lizard.longdata)
lizard.longdata$L

liz.logistic <-  glm(lizard.longdata$L ~ ., family = binomial, data=lizard.longdata)

#effects
logistic.LRT.stats<-Anova(liz.logistic) #default test.stat is LRT, LRT stats 
logistic.LRT.stats
cbind(liz.logistic$coefficients,confint(liz.logistic),exp(confint(liz.logistic)))
anova(liz.logistic)
AIC(liz.logistic)
liz.logistic.sat <-glm(lizard.longdata$L ~ I*P*D*T, family = binomial, data=lizard.longdata)

We use Anova function from the car package to get the LRT stats which is shown below

The effects are shown in the table below for 95% confidence interval. The last two column show exponentiated CI.

Best subset using AIC

We applied stepAIC function on the full variable model which resulted in giving us the same model. In other words, it was the one with the lowest AIC value of 577.62, and a residual deviance of 567.6216.

L ~ I + P + D + T

As we see in the influence plot below the residuals on the best model obtained is much higher than what we get on the loglinear model (Above influence plot for loglinear model).

We infer that this model does not fit the data well.

Final words on Logistic and Log-Linear model equivalency

It would make sense that the two models would be equivalent, we also know that the logistic model doesn’t know interaction terms. However, in this analysis we don’t see an equivalency, as the deviance of the two models are quite different. We have obtained 26.1033 for the best log linear model, and deviance of 555.4047 for the best logistic regression model.

We conclude a log linear model expresses the relationship nicely between the species and its variables that help us discern it.

There are a lot of great topics to delve into when working with categorical data. I will cover them in future posts. I hope this analysis was helpful in understanding how to work with GLMs.

Until next time!

References:

McCullagh, P., Nelder, J.A., 1989. Generalized linear models. London, Chapman & Hall, 511 p.

Agresti, A. 2007. An introduction to categorical data analysis. Wiley.

https://stats.stackexchange.com/questions/26930/residuals-for-logistic-regression-and-cooks-distance

http://users.stat.ufl.edu/~presnell/Links/R-links.shtml

• Home - RDocumentation

• lizards: A Comparison of Site Preferences of Two Species of Lizard in aod: Analysis of Overdispersed Data

In the final part of this series we will compare the results of our modeling work between classical time series models and neural network architectures applied with multivariate data that we previously discussed vs the univariate benchmark models.

If you haven’t read the previous posts, please do take a moment to review them. I have added a brief summary below outlining what was covered along with links to those articles.

In Part 1, we introduced the importance of US unemployment rate forecasting as well as how a multivariate modeling approach can be applied.

We also discussed the various macroeconomic variables used in this analysis as well as an overview of the time series modeling approach.

In Part 2, we discussed important data preparation considerations, as well as exploratory data analysis as it applies to standard time series datasets. We covered time series decomposition, autocorrelation, partial autocorrelations as well as ADF test for stationarity.

In Part 3, we discussed how Vector Autoregressive (VAR) models work. We also covered two neural network architectures including Artificial Neural Network and Recurrent Neural Networks and a discussion on LSTMs. We concluded the discussion in part 3 with model evaluation metrics used for this work.

Model development procedures

For forecasting the unemployment rate, each of the three methods are first used with a univariate setting, followed by multivariate model fitting and forecasting. The model development was done using Python libraries (version 3.8.5). The details on specific packages and functions are listed in the below tables:

Classical Time Series — Univariate analysis

An optimal lag order of 5 was used for univariate model fitting. It is obtained by fitting the model with a max lag of 24 when calling the ar_select_order() function using ‘aic’ as the information critieron. The model is fit using the ARIMA library from statsmodels package and a 12-step forecast is obtained. The summary() function provides the below results for AR(5):

The model fit results are shown in above output for AR(5) model. Lags1, 2 and 5 are found to be statistically significant along with a white noise component sigma2.

Once the model is fit, walk forward multi-step forecasting for a forecast horizon of 12 is done. This is inclusive of all forecast horizons until 12. This forecast is done by providing end of the training period as starting period for forecasting, along with a lag of 5 as the input. Each forecast call returns a forecast for the next 12 periods. The input vector is changed by sliding the input window forward (walk-forward) by one incremental observation at a time and selecting the next 5 observations as input. Each output vector of the 12 forecasted values for the different forecast steps is separately stored in a list.

Mean Absolute Error is calculated by comparing the lists of actual unemployment rate for the various forecast horizon with the lists containing forecast value from above. Time series data for forecast and actuals is plotted as shown below in figure (forecast in blue, and actuals in red).

The validity of the data setup is tested by comparing the results of the AR model developed in this analysis for a single forecast horizon window and comparing it with results obtained in the Federal Reserve paper (Cook & Hall, 2017). The MAE obtained is 11.5 bps (basis points). Results of this test and others described in sections described further are summarized in the results and discussions section.

Classical Time Series — Multivariate analysis

The standard VAR(p) model implementation from statsmodels is not appropriate for non-stationary data, so dataset in this analysis is made stationary. (Vector Autoregressions, 2022). The results of ADF test applied across the entire list of variables was discussed in part 2 of this series. The following variables that are shown (and only the leading and trailing records are displayed) in the full time series data, were used for fitting the VAR model.

Again, max lags of 24 is provided to determine optimal lag, where BIC provides 1 as the optimal lag. For selecting the lag select_order() function from statsmodel package is used. Partial output has been reproduced below. The results show that the unemployment rate and the GDP at lag = 1 are statistically significant to explaining unemployment rate at current time t.

Granger causality

Granger causality test is done using grangercausalitytests function from statsmodels which provides the following results. We pass the value of 1 to maxlag parameter.

The tests indicate that consumer price index growth rate (CPIGrowthRate1d_x), and Money Supply (LogMoneySupply_1d_x) both are positively correlated with Unemployment Rate (UnemploymentRate_y) and granger causes Unemployment Rate. It is important to note that, though the name suggests “Causality” it is not meant to imply causality. The differentiating phrase used is variable x “granger causes” variable y. A better way of thinking about it is that it translates to correlation between the two variables.

The values in this table show the p-values, and, if they are less than the significance level (0.05) then it implies that the coefficients of the corresponding past values is zero, which is to say, the null hypothesis that X does not “granger” cause Y can be rejected.

Classical Time Series — Forecasting

Rolling forecast is applied to get multi step multi output for 12 step horizon using the forecast function from the fitted VAR model. The rolling forecast is obtained in the same way it was done for univariate analysis by creating separate lists to store the actual unemployment rate for various forecast horizons as well as for storing the forecasted unemployment rate fot those forecast horizons. The mean absolute error is then calculated across these lists for each forecast horizon consecutively and reported.

Following results are obtained by calling the mean_absolute_error() function from sklearn.metrics package. This is done by providing the list with forecasted values along with actual values, one list each for the 3,6,9 and 12 step forecast. The MAE were obtained as 11.2 bps, 19.7 bps, 27.7 bps and 35.9 bps respectively. (More summarized comparative views below).

Model development using Neural Networks

A key experimental setup that is replicated across all of the neural network architectures in this analysis across univariate and multivariate analysis is the process of setting up validation data set from the original training dataset, which is from 1963 to 1996. In this dataset, every tenth observation is removed from training data set and passed into a validation data set. Each observation is a set of continuous 36 month data. This dataset is then used to evaluate the performance of the trained model. The testing dataset contains the remainder of the monthly data from 1997 to 2014.

Since this analysis is meant to compare an existing univariate work in a new multivariate setting, we retained the experimental settings similar to univariate work by Cook and Hall. In their work, they use a lag of 36 months of the unemployment rate data for neural networks. Therefore, this analysis also uses lag of 36, specifically for the univariate analysis using ANN. Lastly, they also use the original unemployment rate, the first order difference and the second order difference unemployment rate in the univariate model. The setup for univariate forecasting in this analysis is replicated with their setup.

When working with Neural Networks it is important to standardize the data, particularly when the data is not scaled. If the data is not standardized, attributes that have higher values will take on higher weights in the model incorrectly influencing the model. For this analysis neural network models have been evaluated separately with both standard scaling and min-max scaling. The sklearn scaler object is fit with the training data separately for dependent and independent data. Then as the next step, independent data (x) is transformed. The scaler object is not fit with testing data so as to avoid data leakage.

As discussed in the data preparation section, there are two different types of data transformation applied: (1). Standardization and (2) Min-Max Normalization. Each neural network is trained, evaluated and used for forecasting. The final results are reported based on the best model evaluation metrics across the two types of transformation. When the model is trained, and forecast is generated, the forecast is reverse transformed to original scale of the data for calculating mean absolute error.

When working with machine learning algorithms including deep learning algorithms, time series problem needs to be converted into a supervised learning problem. This is done by taking sequences of data iteratively from the original time series and splitting the time series data into a set of time series data for training, and another set of time series data for testing. It is in this set of training sequences, the sequestering (hold out) operation described above is done for model evaluation.

For LSTM under univariate setting, the number of lags is chosen to be 12. On splitting the training data into training and valiation, X training data is obtained with the shape (344,12,3) with 344 samples, 12 lags, and 3 features. The Y training data is of the shape (344,12) with 344 samples and 12 lags. The testing dataset is setup in the same way without the need to sequester any observation, that provides a X testing data of the shape (194, 12, 3) and Y testing data of the shape (194, 12).

TensorFlow from the keras package in Python is used for implementing the neural network architectures in this analysis. A tensor is a 2D matrix with additional dimensions. A 3D tensor is an appropriate data structure to store time series data. This can include storing the samples, time steps and features, which is useful especially for a multivariate problem as shown in figure below.

Keras model development workflow:

The workflow required for model development using Keras requires is as follows (Chollet, 2018):

1. Define the training data using numpy arrays or tensors.

2. Define a model that is a network of layers which maps the inputs to outputs.

3. Choose a loss function, optimizer and a model evaluation metric for monitoring the model training

4. Loop through the training data by calling fit() method

A Tensorflow model is defined by creating an object of the class Sequential() that creates linearly stacked layers. A LSTM layer is added with 32 nodes, relu activation function along with the specification of the input shape that includes the number of lags, and number of features. This is followed by addition of a Dense (final output) layer with as many nodes as the number of forecast steps. A ModelCheckpoint is created to save the best model information based on lowest validation loss. Finally, the model is compiled by choosing ‘adam’ optimizer and the model is set to be optimized for lowest MAE.

On fitting the LSTM model with 100 epochs, following summary results shows the type of layers is obtained. It shows the two linearly stacked layers LSTM and Dense.

The training loss and validation loss drops in less than 10 epochs, and plateaus after 50 epochs as shown below.

A12-step horizon using the best model was obtained. Which was then then reverse scaled to calculate the error on forecasted data. Following results were obtained for univariate LSTM:

Overall mean absolute error in percent : 0.3539

Overall mean absolute error in basis points : 35.39

The below chart shows the forecasted vs actual value for the 0th step forecast (first month) in the 12 month forecast horizon.

Discussion on Multivariate forecasting with LSTM

The setup for multivariate forecasting for LSTM is the same as that for univariate except for the data setup. The supervised data set for training and testing includes multiple variables as shown below.

The tensors setup for multivariate forecasting are of the following shape

XTrain: (344,12,25)

YTrain: (344,12)

XTest: (194,12,25)

YTest: (194,12)

LSTM model for multivariate forecasting is also setup with two layers as shown below:

The training loss and validation loss are outputted as follows. The loss convergence is similar to the univariate LSTM model training.

The model evaluation error obtained is 0.0292. Again, forecasting with the best LSTM multivariate model, a vector of 12 step forecast is obtained. For illustration the first record of the forecasted values and actual values are reproduced below:

Compared to the univariate LSTM mode the forecast for the first month of the 12 month period shows lesser accuracy (as you will see, not good!). The chart below shows the performance of the LSTM multivariate model after inverse transforming the forecast.

The MAE for the the 12 step forecast horizons is as shown below:

Discussion on Univariate Forecasting with Feed Forward ANN:

For fitting an ANN model using Tensorflow in Keras, the data used for LSTM needs to be reshaped such that the structure of the input data is stored as (samples, features). In this analysis, forecasting using ANNs was done with both min-max scaling and standardization scaling. The model results were better when using standardization for univariate setting, while the results were better when using min-max normalization for multivariate setting.

All the other experimental conditions were retained as before, an input lag of 12 was chosen. Again, two linearly stacked layers were chosen for fitting the ANN. The univariate model output was obtained as follows:

After fitting the model, MAE of validation was obtained as follows:

After fitting with the best model, forecasting the 12-step output, and inverse transforming, the following plot is obtained:

The MAE for the 12 step forecast with the univariate ANN is obtained as shown:

Discussion on Multivariate Feed Forward Artificial Neural Network

This model is also a 2-layer linearly stacked neural network. The model information is provided below.

The test loss and test accuracy is shown as below:

The training loss and validation loss is plotted below:

Again, after forecasting with the best model, inverse transform is done followed by calculation of the MAE. Results obtained are shown below.

The plot for the first month forecast in the 12-month forecast horizon is plotted below.

Of all of the model used so far, it is easy to tell visually that this model performs poorly (of all the other models) developed in this analysis.

Let’s look at how we did across all the models.

Model Evaluation Results

A comparative analysis of the models is tabulated as shown in table below.

The Mean MAE is shown for the VAR, ANN and LSTM models. Each model has an univariate (U) and multivariate (M) result tested on out of sample data from 1997 to 2014.

The last column shows benchmarks obtained from the data published by SPF (Survey of Professional Forecasters) at Philadelphia Federal Reserve Bank. These benchmarks were used for comparison of univariate analysis and forecasting by Cook and Hall (2017). This analysis used the same data and benchmarks as the previous analysis by the Fed Reserve authors which was done using univariate methods.

The below figure shows graphical assessment of the MAE of the different models over the multiple forecast windows when compared to the SPF benchmark. The brown line in the center indicates the SPF forecast. The top two lines indicate the multivariate neural networks. The bottom four lines indicates the univariate AR model, neural networks and the VAR(1) multivariate model. A comparative explanation of models generated from this analysis vis-a-vis the benchmarks follow.

This research Univariate vs Fed Univariate — In this analysis, the AR(5) model compared relatively better than Fed’s Directed AR Model (DARM). It performed just marginally better than SPF’s forecast. It also performs better than benchmark univariate LSTM. While there is not enough information on the model specific hyper tuning parameters used in their univariate LSTM, it is plausible their model could perform equally well under similar parameters. It is also plausible that the univariate AR model from this analysis outperformed the directed AR model because of the usage of optimal lag and usage of first order and second order difference as model inputs. There is no information on how the DARM was setup by Federal Reserve to further comment on causes for performance differences between the two.

This research Multivariate VAR Model Performance — The VAR model from this analysis performed best for the three month forecast period, with comparable performance with the neural network univariate models for six month and nine month forecast period, while its performance was slightly unfavorable compared to the twelve month forecast period. The multivariate VAR(1) model outperformed the univariate AR model across all quarters with diminishing gap in their performance as the number of forecast steps increased. It also performed better than the vanilla LSTM model by Cook & Hall and other Federal Reserve models. Based on these results obtained in the VAR model, it can be concluded that the information contained in the macroeconomic indicators chosen for forecasting unemployment rate have a reasonably strong relationship with unemployment rate, more than the strength of the relationship solely found as a result of autocorrelation of lagged unemployment rate data, at least for shorter forecasting period.

This research Univariate Neural Network models — The univariate ANN and LSTM models performed better than VAR(1) specifically for the 12 step forecast period with comparable performance for three and six step forecast periods. It is possible that a more optimally tuned neural network and other architectures might be able to beat the VAR(1) model not just for the longer step forecast periods, but consistently across all the periods.

This research Multivariate Neural Network models — The neural network models showed promisingly low MAE for validation data. However, both LSTM and ANN fared poorly on the test data. The LSTM model had two to three times the MAE on an average compared to SPF models. This is likely due to reasons such as overfitting and propagating errors due to predicting multiple variables.

The three subsequent tables below show the MAE and Standard Deviation for each models developed in this analysis and they are compared with benchmark MAE and Standard Deviation. The standard deviation was computed for the forecast errors. Three of the models — multivariate VAR(1), univariate LSTM and univariate ANN have a smaller dispersion around the mean when compared to the dispersion of the SPF benchmark model around their respective means. Whereas, the univariate AR(5), multivariate LSTM and multivariate ANN all have a larger dispersion compared to that of the SPF benchmark models. All of models developed in this analysis, including the best performing multivariate VAR(1) model have a larger dispersion around the mean relative to Cook & Hall’s standard deviation around their model means. MAE and SD are in basis points.

Conclusion

One of the questions this research set out to answer was whether adding more variables helps improve model performance. The results established that is true, as shown by the results of the multivariate vector autoregressive model, specifically for shorter forecast windows.

Another question this research sought out to answer was whether using a neural network model can help improve the forecast accuracy. The results provide a mixed answer to this question. The univariate neural network models — both ANN and LSTM did favorably well when compared to Fed’s models. However when more data was introduced into the neural networks, the model performance deteriorated significantly. The neural networks had minimal tuning. It is possible that under the right set of hyper parameters, as well as by using other more recently developed neural network architectures may result in better outcomes for forecasting unemployment rate.

Limitations

One of the limitations of these models is they are not grounded in the economic theories that underpin the relationship between the macroeconomic variables. So, attributing causality from any of the other multivariate features to explain the dependent variable is not feasible. This research can be extended using Structural VAR which allows modeling assumptions of directional relationship between the various variables.

This research also did not consider implementation of cointegration in these models, which in finance and economics is the idea of equilibrium achieved between two different economic variables over a long run (Zivot & Wang, 2003).

Recommendations for further research

There are several avenues that can be explored in the future to extend this work.

1. There is an opportunity to expand the VAR model by including other variables that can explain unemployment rate, variables related to other economic activity such as stock market returns, private investments such as equity financing and many other economic indicators.

2. Explore other recurrent neural network architectures such as Bi-directional LSTM in a multivariate setting.

3. Employ permuting of multivariate combinations as well as hyper parameter optimization for neural network training.

4. There is volatility in the unemployment rate which could be modeled using Generalized Autoregressive Conditional Heteroskedasticity (GARCH) to handle different volatility across different times.

5. The unemployment rate appear to in different regimes across different long-run economic cycles. One option to model this would be to use regime switching using Hidden Markov Models (HMM).

6. Lastly, and importantly, with recent advancements in transformer architectures and its application to time series data, there is an opportunity to consider implementing these architectures to forecasting unemployment rate.

In closing, this was an interesting and very worthwhile exercise in applying academic rigor to real world financial forecasting application. This work was the basis of my talk at ASA (American Statistical Association) Symposium on Data Science and Statistics at Pittsburgh, PA (June ’22)

I know there are many other considerations I could have included in the analysis. I welcome your comments/feedback, and look forward to being in broader conversation with you on application of data science techniques.

Acknowledgement

A shout out and acknowledgement to all the researchers, educators and authors who spent countless hours building and teaching this knowledge base over the many decades. It is only out of their efforts do data science practioners like myself are able to do what we do. I am grateful to all of you.

Until next time!

References:

Chollet, F. (2018). Deep Learning with Python. NY: Manning.

Cook, T., & Hall, A. S. (2017). Macroeconomic Indicator Forecasting with Deep Neural Networks. Retrieved from dx.doi.org: https://dx.doi.org/10.18651/RWP2017-11

Karpathy, A. (2015). Retrieved from http://karpathy.github.io/2015/05/21/rnn-effectiveness/

Montgomery, A., & Zarnowitz, V. (1998). Forecasting the US Unemployment Rate. Journal of the American Statistical Association, 478.

SPF. (2022). Survey of Professional Forecasters. Retrieved from Federal Reserve Bank Philadephia: https://www.philadelphiafed.org/surveys-and-data/real-time-data-research/survey-of-professional-forecasters

Vector Autoregressions. (2022). Retrieved from Statsmodels: https://www.statsmodels.org/dev/vector_ar.html

Zivot, E., & Wang, J. (2003). Vector Autoregressive Models for Multivariate Time Series. In E. Zivot, & J. Wang, Modeling Financial Time Series with S-Plus®. New York, NY, USA: Springer. doi:https://doi.org/10.1007/978-0-387-21763-5_11

In my previous posts on this topic we covered two parts so far.

In Part 1, we discussed the importance and relevance of US unemployment rate forecasting and why a multivariate modeling approach is necessary.

We also discussed the various macroeconomic variables used in this analysis as well as an overview of the time series modeling approach.

In Part 2, we discussed the data prep steps as well as exploratory data analysis as it applies to standard time series datasets including time series decomposition, autocorrelation, partial autocorrelations as well as test for stationarity.

In this post, we will begin the discussion with step 5 of the 6 stepped process we outlined in part 1 of this series.

Let’s go to step 5:

5. Choosing and Fitting Models: In this step, models are trained using the training data and evaluated using validation data. These are chosen based on assumptions the researchers make that the model satisfies. Models are built using multiple parameter (or, hyperparameter values) to build the most efficacious model using historical data. In this post we will discuss Vector Autoregressive model as well as two Neural Network architectures.

Vector Autoregression (VAR) Models

A vector autoregression (VAR) model is a flexible time series model that is used for multivariate analysis. It is an extension of the univariate Auto regressive (AR) model and works with multivariate time series.

A VAR model contains a system of equations of n distinct, stationary response variables as linear functions of lagged responses and other terms. VAR models are also characterized p lags of all variables in the system, denoted as a VAR(p) model.

A bivariate VAR(2) model with time series variables y1t and y2t can be represented as follows:

VAR models belong to a class of multivariate linear time series models called vector autoregression moving average (VARMA) models.

Interpretation of a general VAR(p) model is difficult due to complex interactions between the variables in the model. So the model is described using different types of structural analysis summaries including Granger causality, Impulse response functions and forecast error variance decomposition. This analysis includes Granger causality of other predictor time series data on the unemployment data. Granger causality is covered in part 4 of this series.

Whu use Vector Autoregressive Models?

VAR models are linear models. However, non-linearities can be modeled by applying various adjustments including number of lags to include for each time series, which co-variate to use, incorporating moving-average components and accommodating co-integration (Hyndman & Athanasopoulos, 2021).

The number of lags that are appropriate for forecasting the dependent variable is obtained by minimizing information criterion measures such as Akaike Information Criterion (AIC), Bayes Information Criterion (BIC), Hannan-Quinn Information Criterion (HQIC) and Final Prediction Error (FPE) (Lower, Eric, 2021). The lag that had the smallest information criterion value across all of these measures was chosen to provide a parsimonious model (ResearchGate, 2022). Deciding what variables to use can be done by applying Granger Causality tests. Granger Causality only provides the correlation across the variables and is not a true test of causality. In this analysis, the number of variables used for forecasting is small, so all the variables that were sourced were used for forecasting with VAR. Vector Autoregressive models have been proven to be useful for describing and forecasting economic and financial time series data. Similar to DSGE described above, VAR models are also used for structural inference and policy analysis (Zivot & Wang, 2003).

Artificial Neural Networks

The fundamental pattern based on which deep learning models are built is a perceptron. A perceptron has three set of nodes — (1) input nodes, (2) computational nodes, and (3) output nodes. Each set is called a layer. What uniquely differentiates a perceptron is that it has a single computational layer. This computational layer is also called the hidden layer.

The perceptron model uses a linear combination of inputs with some weights. It then applies an activation function to produce an estimated output. This model takes input of k lags of time series x. And it estimates the value of the time series at a period t+n. The values ωo to ωk are the weights associated with each of the inputs. These weights are calculated using a cost function that minimizes the error, MSE (mean squared error). The perceptron without the activation function is nothing but a linear regression of the inputs with the weights. It is the transformation applied by the activation function that allows non-linearity of the underlying relationship to be modeled.

When there is one or more than one hidden layers wherein the information flows from previous layer to the next layer, the architecture is called Multilayered Feed Forward Neural Network.

The weights is what is used to determine which signal, or input should pass through. The weights are assigned randomly initially. An activation function (sigmoid, tangent hyperbolic (tanh), rectified linear unit (relu), or softmax) is applied to the linear combination of the input nodes. The result is passed to next layers, this step is repeated until the output layer which generates a vector of probabilities for various outputs. The resulting prediction is compared with actual values to determine error and this “feedback” is backpropagated to the nodes to readjust the weights and reapply the activation all over again. This is re-iterated until the cost function or loss function is minimized.

Why use ANN?

For this specific problem, non-linear relationship is assumed to be exist between the predictors and the target variables. ANNs are known to be effective in modeling non-linear relationships. ANNs were also found to be significantly better in forecasting unemployment rates when compared to autoregressive models. This however holds true when there is sufficient data, and the neural networks have an appropriate architecture and are not simplistic such as having a single hidden layer. In this analysis the neural network has a dense hidden layer. Further, the data obtained for this analysis is assumed to be sufficient to outperform autoregressive model. (Mulaudzi & Ajoodha, 2021)

Recurrent Neural Networks

A recurrent neural network (RNN) is different from a plain vanilla neural network as discussed in the previous section in that a plain vanilla neural network accepts an input X that is of fixed length, and outputs a Y that is also of a fixed length. RNNs on the other hand can accept a sequence of inputs, more specifically, a sequence of vectors and output a sequence of vectors (Karpathy, 2015). In RNNs, the hidden layers store the information captured in previous stages of sequential data. The term ‘recurrent’ in RNNs is used because these neural networks perform the same task for every element of the sequence, specifically that of utilizing previously seen sequence to predict future unseen sequential data.

In Figure 3 below, the right hand side shows X at time 0 to time t. Each neuron or group of neurons, A produces a cell state h that is passed on to the next set of neurons. The cell state of the neuron at time t is remembered as ht. The left hand side shows the recurrence relationship of the pattern where h is updated at each time step. The right hand side shows the recurrence relationship in an unrolled view.

For an excellent reading on RNNs and LSTM, I recommend seeing Colah’s blog. Link to the original article is in references section.

Back to the discussion — The RNN can be represented mathematically in two steps occurring repeatedly across each time step. In the forward pass the following steps are done.

a. Calculate the hidden state: This is obtained by applying an activation function to the current input and previous state. The hidden state is given by

The tanh function applies a non-linearity and pushes the activation output to be in the range from -1 to +1.

b. Calculate the output: The output for time step t, formally y-hat of t, is obtained by

The learning that occurs in the RNN is really the calculation of the optimal weights across the three matrices that minimize the loss function. The matrices, again are:

Problem with RNNs

However, there is a drawback with a typical generic RNN that these networks remember only a few earlier steps in the sequence and thus are not suitable to memorizing longer sequences of data. RNNs when trained using backpropagation suffer from the vanishing gradient problem, wherein the gradients become so small that there is no real learning that can occur from inputs that were provided in the distant past time step. A class of RNN Long Short-Term Memory (LSTM) recurrent network overcomes this challenge.

LSTM is a special kind of RNNs that are designed to memorize long-term dependency in a sequence of data. There are four layers within each cell of LSTM as shown in Figure 4 whereas each cell of RNN has only one layer as shown in Figure 5.

There are two types of information passing that occurs in LSTM. The top line shown in the cell in Figure 5 represents the information about the cell state. Information can be added to or removed from the cell state line which is controlled using gates.

The first layer uses a sigmoid activation to select between fully remembering or fully forgetting the information from the prior cell state. This is also called the forget gate.

The second layer also uses a sigmoid activation (also called as input gate layer), is used to select which information should be updated. The third layer uses a hyperbolic tangent activation that creates a vector of new possible values, that can be added to the cell state.

The next step is to update the new cell state by combining operations to forget what was determined to be forgotten and add new candidate value. The final step is to output which is based on a combination of sigmoid and tanh activations. The sigmoid activation determines which part of the cell state to output (sigmoid chooses between 0 and 1), whereas the tanh activation will finally push the output (tanh chooses between +1 and -1)

Why use LSTM?

LSTMs are essentially NNs that incorporate feedback loops. The architecture of LSTM described above allows for its use in sequence modeling. Specially, features of the data from the long running memory, as well as short term memory where most recent sequence of information is utilized. These models outperformed autoregressive models in an experiment conducted to forecast unemployment rates in the US (Mulaudzi & Ajoodha, 2021).

Next we’ll look at the setup for using the forecasting models and evaluating it.

6. Using and Evaluating a Forecasting Model:

The last step is to make forecasts and evaluate the performance of those forecasts. The model evaluation is performed to determine the efficacy of the models. In this analysis, back-testing is done with historical data.

Each of the neural network based multivariate model is compared with its corresponding univariate model. The multivariate models are also compared against one another. Finally, each of the multivariate model is compared with the Survey of Professional Forecasters (SPF) benchmark model data provided by Federal Reserve (Cook & Hall, 2017)

In traditional data mining methods when working with data that is not time series, k-fold cross validation is used to systematically divide the samples into k groups, where each group (in the k groups) is iteratively held out for model validation. This works for datasets that are not temporal in nature (they have no time dependency), therefore, in those cases each observation is independent.

Time series data has a unique characteristic in that there is a temporal dependence of variables. Machine Learning techniques require setting up a time series data as a supervised data set. In addition, this requires the splitting of the data for training the model and evaluation to be done differently from traditional machine learning methods.

The models are compared against one another using the metric Mean Absolute Error (MAE) and standard deviation (SD). This is a well-known criterion for comparing forecasting accuracy of time series models. (Katris, 2019)

Where, yt is the actual unemployment rate for a period, and y-hat t is the forecasted unemployment rate for the same period, and N is the number of forecasts.

As this research is an extension of previous work done by researchers Cook and Hall (2017), this analysis uses the same feature UNRATE (unemployment rate data as made available by the Federal Reserve) for the same periods and partitioned the data for training and validation (1963 to 1996) as well as testing (1997 to 2014). The results obtained from this analysis were compared with the researchers results as well as other benchmarks referred to in the original paper. The creation of the validation data was done in the same manner as their paper by sequestering every 10th observation of the training data into a validation data set.

At this point we have all the elements ready for model building and evaluation. And that is what we will discuss in the next post.

References:

Colah’s blog: https://colah.github.io/posts/2015-08-Understanding-LSTMs/

Cook, T., & Hall, A. S. (2017). Macroeconomic Indicator Forecasting with Deep Neural Networks. Retrieved from dx.doi.org: https://dx.doi.org/10.18651/RWP2017-11

FRED. (2021–2022). Fed Reserve Economic Research. Retrieved from Fed Reserve Economic Data: https://fred.stlouisfed.org/

Hyndman, R., & Athanasopoulos, G. (2021). https://otexts.com/fpp3/. OTexts.

Karpathy, A. (2015). Retrieved from http://karpathy.github.io/2015/05/21/rnn-effectiveness/

Katris, C. (2019). Prediction of Unemployment Rates with Time Series and Machine Learning Techniques. Computational Economics, 682.

Lower, Eric. (2021). Introduction to the Fundamentals of Vector Autoregressive Models. Retrieved from Aptech: https://www.aptech.com/blog/introduction-to-the-fundamentals-of-vector-autoregressive-models/

Montgomery, A., & Zarnowitz, V. (1998). Forecasting the US Unemployment Rate. Journal of the American Statistical Association, 478.

Mulaudzi, R., & Ajoodha, R. (2021). Application of Deep Learning to Forecast the South African Unemployment Rate: A Multivariate Approach. IEEE Xplore.

ResearchGate. (2022). Retrieved from researchgate.net: https://www.researchgate.net/post/How-do-you-choose-the-optimal-laglength-in-a-time-series

Vector Autoregressions. (2022). Retrieved from Statsmodels: https://www.statsmodels.org/dev/vector_ar.html

Zivot, E., & Wang, J. (2003). Vector Autoregressive Models for Multivariate Time Series. In E. Zivot, & J. Wang, Modeling Financial Time Series with S-Plus®. New York, NY, USA: Springer. doi:https://doi.org/10.1007/978-0-387-21763-5_11

In the previous post (linked here: Part 1) we discussed the importance of unemployment rate forecasting. We also framed how we approach the analysis and time series modeling. So far we covered the problem definition and the data sources used.

In this post, we’ll look at few relevant python code, charts and data snippets.

And importantly, we will cover the data preparation phase and look at the exploratory data analysis which has unique set of analytical tools to understanding the time series data.

So, continuing on …

Here are the libraries used for this analysis. Statsmodels is a very useful package for working with classical time series methods.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from statsmodels.tsa.seasonal import seasonal_decomposepu
from statsmodels.tsa.stattools import adfuller
from statsmodels.graphics.tsaplots import plot_acf
from statsmodels.graphics.tsaplots import acf
from statsmodels.graphics.tsaplots import plot_pacf
from statsmodels.tsa.stattools import grangercausalitytests
from statsmodels.tsa.ar_model import AutoReg
import statsmodels.api as sm 

Here is a snapshot of a few sample records from the multivariate time series.

Using seaborn for univariate seasonal plots:

plt.subplots(figsize=(20,5))
sns.boxplot(x=ue_seasonal_plotdf['month'],y=ue_seasonal_plotdf['UnemploymentRate'])py

This gives us the seasonal plot by month, followed by seasonal plot by quarter. Leads us to believe nothing quite extraordinarily different across months or quarters.

3. Data Preparation Phase — Time series data has a unique characteristic in that there is a temporal dependence of variables. A variable at a current time has dependence on the variable at one or more than one periods.

A time series has three “systematic” components that can be described and modeled. These are ’base level’, ’trend’ and ’seasonality’, plus one “non-systematic” component called ’noise’. The base level is the mean value in the series. The time series is said to have a trend when the time series values increase or decrease over time (positive or negative slope). When there is a pattern repeated at regular intervals, the time series is said to have seasonality. Any random variations is the noise. Every time series data is a combination of these four components. While, base level and noise always exist in a time series data, trend and seasonality can optionally exist.

There are several factors that need to be considered due to the nature of time series data which includes preprocessing and preparation of data for the analysis of such data. There are several transformations that are applied to make the data work with the methods used in the analysis and forecasting of the time series data. Additional feature engineering is done using these transformations.

a. Box Cox / Log Transformation — This removes the changing variance over time. This is done using np.log() function using numpy library. This is applied to variables MoneySupply, ProducerPriceIndex, CandILoans and CRELoans for training and forecasting with the neural network models.

b. Differencing — This is applied to de-trend the data and to make the data stationary. Differencing is used to attain stationary time series which is a model assumption for working with Vector Autoregressive (VAR) models. The differencing involves using the difference in values over two consecutive periods of time. This is done using diff() function from the pandas library to calculate a differenced time series value.

c. Normalizing — Normalization of data makes value of each time series data point to be between 0 and 1. This data transformation is required to work with neural networks. This is done using MinMaxScaler() function from the sklearn.preprocessing library.

d. Standardizing — Standardization of data normalizes value of each time series data point on a z-scale with a mean of 0 and standard deviation of 1. This was done using StandardScaler() function from the sklearn.preprocessing library. This data transformation is required to work with neural networks.

For final forecasting of future (test) data, the normalization and standardization are reversed to compute the error. Both normalization and standardization is applied as part of data transformation to train and forecast using neural networks.

4. Preliminary (Exploratory Data) Analysis: The datasets for this analysis were sourced from the Federal Reserve database that did not have issues with missing data. The analysis is done separately for univariate setting, followed by multivariate setting.

Univariate Analysis

As shown in Figure 1 the unemployment rate is stationary (constant mean) with varying volatility over the range of period shown here.

Figure 2 shows the distribution of unemployment rate for this period, it ranges from 3.5% to over 10.5% with the distribution skewed to the right. This data doesn’t quite have a normal distribution.

Figure 3 shows a comparative view of the unemployment rate and its first order difference (1D) computed by calculating the difference in consecutive values of the unemployment rate data).

Figure 4 shows the distribution of the first order differenced unemployment rate. This data is centered around zero.

Time Series Decomposition

To improve forecasting accuracy time series data often requires to be analyzed by decomposing it to understand the underlying patterns.

There are three components: trend, seasonality and residuals. Additive decomposition can be expressed as:

Unemployment Rate = Trend + Seasonality + Residuals

In Figure 5 below, the top chart shows the original time series (same as Figure 1 above). This is decomposed into the three components trend, seasonality and the residuals, which is whatever remains after the trend and seasonality have been separated from the time series. This time series does not have a trend, but it does have a strong seasonality to it. The residuals are dispersed around the mean of zero.

Figure 6 shows a histogram of residuals which shows that the residuals are normally distributed across the mean zero.

Autocorrelation and Partial Autocorrelation

Autocorrelation shows the extent of linear relationship between the value of a variable in a time series at time t and the value of variable in that time series at a time t+h or t-h. A positive correlation indicates the value of the variables move in the same direction, whereas a negative correlation indicates the value of the variables move in opposite directions. When autocorrelation is zero, the temporal dependency is difficult to establish. Autocorrelation plots help understand this temporal relationship. In Figure 7 shown below, the x axis shows number of lags, and the y axis shows the autocorrelation which is normalized between the values of 1 and -1. The orange band shows the 95% confidence interval. This chart shows that unemployment rate autocorrelation can be seen until 24 lags (24 months).

Not only is there linear relationship between xt and xt+h or xt-h there is linear relationship between xt and the lags leading upto xt+h or xt-h. In other words, xt also has dependency on intermediate lags, therefore autocorrelation is not the correct measure of mutual correlation between xt and xt+h or xt-h. This requires removing dependency of the intermediate lags. This is done using partial autocorrelation function. Figure 8 shows a partial autocorrelation chart which shows temporal dependence of unemployment rate at a time t with that of t-1 or lag=1 with some weak dependency shown between lags 2 to 5.

Test for Stationarity

One important assumption that is made for inferential statistics to be reliable is that the population from which a sample is drawn does not undergo any fundamental change over the individual samples or over the time during which the samples are collected. This assumption is important for the inferential statistics to be reliably representing samples not previously seen before. This assumption also applies to time series data (Pal & Prakash, 2017). This assumption in time series is called stationarity, and it implies that the mean, variance, and autocorrelation are constant, and so these can be used for future occurrences of the time series.

Augmented Dickey-Fuller (ADF) tests are used for detecting stationarity in time series data. On applying ADF tests on Unemployment Rate the p-value of the ADF test was found to be 0.033. And, since the p-value for Augmented Dickey-Fuller test is < 0.05, the null hypothesis that the time series is non-stationery is rejected, concluding that the unemployment time series is stationery. Detailed test results of ADF test on all the variables is reproduced in table below. ADF Tests (H0: time series is non-stationary) are assessed at a 5% significance level on original data after differencing (1d) used in unemployment rate analysis and forecasting are shown below. The p-values are obtained for 4 significant digits.

Multivariate Data Analysis

Figure 9 shows all of the original macroeconomic indicators that are used for forecasting unemployment rate. Each of these indicators have been plotted for the same historic period this analysis was done for. A brief discussion of interesting observations for each of the variable follows.

The fed funds rate which is the interest rate that banks charge each other for overnight borrowings was near zero during the financial crisis of 2008 and following that for few years. Producer Price Index (Supplier price inflation), Money Supply (Amount of liquidity provided by Treasury) and Commercial Loan Activities (Commercial & Industrial loans, and CRE loans), are not stationary as determined by ADF tests. These four indicators were differenced to make them stationary.

Additionally, there was differencing applied not only across a single month, but also over a three month period as shown in Figure 10. This was done so as to understand the pattern in the data consistent with methods used for financial data analysis. Financial data is not only evaluated month over month, but also over key recurring periods for financial reporting such as quarter over quarter change, year over year change. This analysis was done using only the three month rate change for the purpose of exploratory analysis. It was not extended for use with any models in this analysis so as to keep the scope of variables to a limited size.

This concludes the exploratory data analysis.

In part 3 we will cover autoregressive methods for both univariate and multivariate analysis.

Until next time.

References:

Cook, T., & Hall, A. S. (2017). Macroeconomic Indicator Forecasting with Deep Neural Networks. Retrieved from dx.doi.org: https://dx.doi.org/10.18651/RWP2017-11

Pal, A., & Prakash, P. (2017). Practical Time Series Analysis. Birmingham: Packt.

Hyndman, R., & Athanasopoulos, G. (2021). https://otexts.com/fpp3/. OTexts.

The unemployment rate is an important macroeconomic indicator that is monitored by US government agencies for the purpose of ensuring the proper functioning of the overall economy. Since the unemployment rate is the measure of joblessness in the economy, it is necessary to build better forecast models for unemployment. These models are also used by the government to implement policy changes to increase employment opportunities and reduce financial hardship on the unemployed during recession.

In this multi-part article, I will discuss the usage of classical time series methods such as Vector Autoregressive Models and different neural network architectures, specifically, Feed Forward Artificial Neural Networks and Recurrent Neural Networks, specifically, Long Short-term Memory Networks.

With these methods, we assess the improvement in forecasting of unemployment rates against their univariate time series equivalent as well as a benchmark model used by the Federal Reserve. These multivariate methods consider historic macroeconomic variables listed below. Explanations for these follow in subsequent sections.

Gross Dometic Product,

Inflation,

Federal Funds Rate,

Commercial Loan Activities, and,

Money Supply (Liquidity).

We further compare models across forecasts over the quarters of a year using mean absolute error and standard deviation. On these metrics multivariate autoregressive model outperformed all the models for shorter forecast horizons, while the univariate neural networks performed better for longer forecast horizons.

What is the Unemployment Rate?

The unemployment rate is defined by the Federal Reserve as follows: “The unemployment rate represents the number of unemployed as a percentage of the labor force.

The Bureau of Labor Statistics (BLS) releases this data every month.

Why is it important to accurately forecast the Unemployent Rate?

The unemployment rate not only gives a measure of joblessness, but also is an indicator of economic growth. This is a lagging indicator and therefore is used to measure the impact of recession whether it is just beginning or is in the decline. It also provides a confirmation of the state of economy when evaluated in combination with other macroeconomic variables. When the unemployment rate increases, as it did during the last financial crisis in 2008, with an unemployment rate of 7 to 8%, which peaked at 10%, the government intervenes by stimulating the economy through a myriad of policy implementation including adding unemployment benefits, adding liquidity into the economy — ‘quantitative easing’, lowering interest rates, and, lowering tax rates to allow access to capital for households and businesses as well as introducing other government spending programs to increase employment opportunities.

Key questions this modeling and analysis is trying to answer:

1. Does including multivariate data improve model performance relative to univariate models?

2. Which multivariate models improve model performance?

3. How do these multivariate models compare with a benchmark consensus forecast from Survey of Professional Forecasters (SPF) at Federal Reserve Bank at Philadelphia.

This article discusses an extension of research conducted by the Federal Reserve at Kansas City (Cook & Hall, 2017), specifically, modeling Unemployment Rate as an univariate time series problem using deep learning architectures.

Univariate time series forecasting of the US unemployment rate do not accurately represent asymmetries which arise from unemployment rate moving countercyclically up during economic contractions and downward in expansions.

The unemployment rate has an inherent contemporaneous dependency of many of the macroeconomic factors. These factors are considered as multiple features, or multiple time series in addition to the time lag (auto-regression) of the dependent variable for forecasting the unemployment rate. Fed Researchers noted that adding more data is expected to improve performance than using a univariate model.

Modeling Approach:

Due to the temporal relationship in the data, CRISP-DM methodology, which is the standard data mining framework, does not work well here.

This analysis is performed using a standardized approach used for time series data (Hyndman & Athanasopoulos, 2021). This original method was enhanced by adding a step for data processing as an explicit step to get the data ready for analysis and modeling. This consists of a six step process:

1. Problem Definition

2. Data Gathering

3. Data Preparation

4. Preliminary (Exploratory) Analysis

5. Choosing and fitting model

6. Evaluating model

Each step is elaborated below:

1. Problem Definition:

As stated, the main objective of this analysis is to learn if adding relevant variables can improve forecasting accuracy of the US civilian unemployment rate. This data is a time series. Problem definition is generally considered the most difficult part of the process due to the subjectivity of the nature of the problem defined. It is important to know who will be using the forecast and its intended usage. In this analysis, the forecasts are being generated for the purpose of answering important questions stated in the three questions described under the Key Question header above.

2. Gathering Information:

This involved collecting historical data for the purpose of analysis and modeling. Additionally, in this phase, supporting information and knowledge base was applied to interpret the information as well as to understand the analysis. The data source for this analysis leverages publications by the Federal Reserve. Federal Reserve Economic Database (FRED, 2021–2022) is the source used for unemployment data as well as for sourcing other data for this analysis. Table 1 shows the time series data are being used for the multivariate analysis.

All the data sets are produced monthly and are not adjusted, except for the GDP which is seasonally adjusted as well as Unemployment Rate, the target variable which is also seasonally adjusted. Below is a brief description of each of the variables and how it is related to unemployment.

a. Civilian Unemployment Rate (UNRATE): This is the target variable. The unemployment rate represents the number of unemployed as a percentage of the labor force. This includes individuals 16 and above in the 50 US states. It excludes active-duty personnel. This is the variable used for forecasting in this analysis.

b. Fed Funds Rate (FEDFUNDS): The federal funds rate is the interest rate at which depository institutions trade federal funds (balances held at Federal Reserve Banks) with each other overnight. When a depository institution has surplus balances in its reserve account, it lends to other banks in need of larger balances. In simpler terms, a bank with excess cash, which is often referred to as liquidity, will lend to another bank that needs to quickly raise liquidity.

The federal funds rate is the central interest rate in the U.S. financial market. It influences other interest rates such as the prime rate, which is the rate banks charge their customers with higher credit ratings. Additionally, the federal funds rate indirectly influences longer term interest rates such as mortgages, loans, and savings, all of which are very important to consumer wealth and confidence according to the Board of Governors of the Federal Reserve System. (FRED, 2021–2022)

c. Money Supply (M2NS): This is one of the measurements (of the three — M1, M2, M3) of United States money supply, also known as the money aggregates. M1 includes money that is in circulation at a given time including checkable deposits in banks, whereas M2 includes M1 plus saving deposits (less than $100,000), and money market mutual funds. M3 includes M2 plus large time (fixed) deposits. The reason M2 is chosen as it is the most watched indicator of money supply and future inflation. The less the money supply, the less liquidity in the system. Optimal level of liquidity is important for access to capital for businesses to continue and expand their operations which further ensures unemployment rates are kept in check.

d. Producer Price Index (PPIACO): This measure provides information on the price index similar to CPI which is generally used for consumer price index (inflation as more commonly known). PPIACO is used as a leading indicator to predict consumer price increases, as it follows a producer price increase. When PPI increases, the cost of raw materials increases and therefore cost of doing business increases. When this price increase is not passed down to the end consumer or is marginally passed down, employers bear the brunt of the cost pressures, putting a risk to employment for the workforce.

e. Inflation (CPALTT01USM657N): This represents CPI (consumer price index) described in section d above. When CPI increases, demand for certain type of goods, services contract. Sustained inflation without intervention from the central bank and Fed can further hurt business sales, and therefore margins, increasing unemployment risk as inflation combined with economic slowdown continues over longer term.

f. GDP Growth (BBKMGDP): Gross Domestic Product (GDP) is the measure of economic activity of a country. It is measured as the total market value of goods and services produced by an economy during a given period. GDP is published only quarterly. An alternative GDP measure is being used here referred to as Brave-Butters-Kelley Monthly GDP. This measure is published by the Fed as a monthly time series data. All data used in this analysis is monthly.

g. Commercial & Industrial Business Loan activity (BUSLOANSNSA): This measure provides commercial lending and is indicator of capital usage by the businesses. A strong loan demand is indicative of demand for labor and therefore a lower unemployment rate.

h. Commercial Real Estate Business Loan activity (REALLNNSA): This measure is similar to the commercial lending activity, except it is specifically for commercial real estate which includes physical properties used for commercial purposes. In an expanding economy with newer offices, facilities, and factories, jobs are expected to be increasing reducing unemployment rate. Both g and h are leading indicators of unemployment rate.

The last two variables described above for business loan activity was not actively found to be used in macroeconomic forecasting literature. However, based on economic theories, we understand that access to capital through loans helps companies plan for growth which requires hiring workforce among other investments, and, therefore higher the loan activity under a rising GDP environment, there is an expectation that the employment rate will be higher. Consequently, the unemployment rate will be lower. After conducting exploratory data analysis, the last two variables were subsequently dropped from modeling.

In part 2 of this series, we will continue the article starting with the data preparation needed for this analysis and modeling.

Until next time.

References:

Cook, T., & Hall, A. S. (2017). Macroeconomic Indicator Forecasting with Deep Neural Networks. Retrieved from dx.doi.org: https://dx.doi.org/10.18651/RWP2017-11

FRED. (2021–2022). Fed Reserve Economic Research. Retrieved from Fed Reserve Economic Data: https://fred.stlouisfed.org/

Hyndman, R., & Athanasopoulos, G. (2021). https://otexts.com/fpp3/. OTexts.

Karpathy, A. (2015). Retrieved from http://karpathy.github.io/2015/05/21/rnn-effectiveness/

Katris, C. (2019). Prediction of Unemployment Rates with Time Series and Machine Learning Techniques. Computational Economics, 682.

Lower, Eric. (2021). Introduction to the Fundamentals of Vector Autoregressive Models. Retrieved from Aptech: https://www.aptech.com/blog/introduction-to-the-fundamentals-of-vector-autoregressive-models/

Montgomery, A., & Zarnowitz, V. (1998). Forecasting the US Unemployment Rate. Journal of the American Statistical Association, 478.

SPF. (2022). Survey of Professional Forecasters. Retrieved from Federal Reserve Bank Philadephia: https://www.philadelphiafed.org/surveys-and-data/real-time-data-research/survey-of-professional-forecasters