AI Roles: Modifier
Exploring what's possible
Observation is a passive science, experimentation an active science.
-Claude Bernard
There's a lot we don't know about AI and machine learning. While we build systems with these methods, it tends to be based on empirical understanding. If we want to design better systems, we need an understanding of what's possible. How can I transform a model to be useful? How can I manipulate a component to do something other people don't know about or think was possible? How do I structure inputs to get consistent outputs? That's where the modifier role on an AI team comes in.
In my previous post on AI Team Roles, I described the modifier role as follows:
Modifiers: Responsible for manipulating components to get the desired output. A more classic example would be those who focus on data transformations and hyperparameter tuning in machine learning models. A more recent example would be prompt engineering. The focus is on getting the most out of each component within a system.
Let's understand the role of modifiers more. We'll look at the various responsibilities of a modifier and how to differentiate between a good and bad modifier.
The Why
There's an increasing rate of development of new AI/ML tools and models, which limits the amount of time that can be spent to understand the potential of each model or tool. If you're solving a problem or building a system with AI/ML, you need to understand how to manipulate the components to achieve what you want. What you really want to know is, "can we accomplish X with model Y?". That's where modifiers come in. Modifiers are part hacker, part mad scientist who understand how to modify systems, models, and inputs to get the desired outputs they want. The goal of the modifier is to figure out what's possible with a given tool or model and use those methods to build a better system. They have a mental framework for understanding how to quickly play and experiment with a new tool to determine if it can be modified to get the outputs they desire.
In some instances, they will find secrets that allow them to build systems that others can't. Take for instance
on getting GPT3 to consistently output a valid JSON, a data structure that's very common in modern applications. At the time, people thought of generative AI models as providing outputs in a question/answer type format or generating lots of options. Most didn't think you could consistently force a very tight structure on the outputs.
Figure 1. The Prompt to Rule All Prompts
He called it The Prompt to Rule All Prompts. While people were still figuring out how to use ChatGPT, Noah had figured out how to get the technology underlying ChatGPT to provide structured outputs, used in typical computing applications, in order to be fed into a larger system. That's the heart of a modifier, experimenting at the edge to show what's possible while still fitting the result into a larger goal.
Concerns
You've seen the modifier activity in other fields, from individuals who have complete mastery and skill with their tools. These are individuals who make you say "I didn't even think that was possible". For instance, look at this excavator climbing a tower on its own.
Unfortunately, not everything modifiers do will be mind-blowing. However, they have a mindset to continuously sharpen their skills through experimentation which enables them to do things others aren't thinking about or don't think are possible. Modifiers are all about exploration and ensuring consistency of results. To that end, some of the questions that modifiers should be asking are:
How do I change the input so that the output looks like this?
How should I change the parameters to achieve X?
How can I run X experiments to find out Y?
What needs to happen to consistently get this required output?
How does this tool behave? What are the edge cases?
What's possible here?
What can this tool not do?
During the product life cycle, modifiers should be engaged at the proof of concept and performance tuning stages. During the proof of concept stage, modifiers are engaged to figure out what's possible with the required system components and to determine what components should be used given their understanding from experiments. Modifiers provide a path forward for getting a proof of concept to work correctly. At the performance tuning stages, modifiers are involved to get system components to consistently output the required results.
Activities
How should a modifier effectively spend their time? Their main deliverables are methods, procedures, experiments, data transformations, and model tuning outputs. The majority of their time should be spent experimenting. Experimenting with new tools, experimenting to understand how to use a tool effectively, and experimenting with what's possible. Across the roles, they are most likely seen as expert wielders who know where each model is strong and weak, and where the holes are. For fun, they might find ways to get car dealers to sell a car for $1 through their chat interface.
Figure 2. Convincing a chatbot to sell a car for $1
A non-comprehensive list of activities modifiers perform includes:
Experimenting to see how far they can push a model for performance or modified outputs
Experimenting to ensure consistent results
Developing tools to let them understand differences between models
Developing tools to automate their experimentation
Sharing knowledge to show what's possible with a given tool or model
Figuring out how to get best in class results with what’s available
Understand the limits of performance for a given tool or model
Understanding how to change data or inputs to get better outputs
What does good look like?
If you were in the head of a modifier, you would probably hear a continuous stream of questions and comments on the order of “let me try this”, “I wonder what happens if I do this”, and “oh that’s interesting”. Good modifiers are continually questioning the world and trying to figure out if they can make something work in an unknown or unintended way. They are continuously experimenting with new tools and should be one of the first team members to show what they've found.
Good modifiers have the ability to meld with and get into the "mind" of a model or tool, particularly the stochastic variety, and understand all of their quirks. This gives them a mental framework to group how different tools work, even those that are adjacent or unrelated. Good modifiers should be able to take any new tool or model they haven't touched before and quickly (much faster than most) figure out how to manipulate it to get the required results. From this skill, they tend to be optimistic about what they can squeeze out of something.
Themes
As modifiers practice their function, there are several themes they should think about to be successful. These themes are operating principles to get one in the right mindset for the role. While each modifier will develop their own working themes and principles, the list below is a good starting point.
Constant experimentation. If you want to understand how something works, you need to continually test it. If you want to find all the quirks, you need to test even more.
Playfulness. Running lots of experiments means a lot of failures will occur. Treating the activity like play, allows one to be more creative and brush off a continual sense of failure with ease.
Translation and transformation. Each model or tool requires a different language be spoken to it, even though they rely on similar concepts. Understanding how to be a "model whisperer" to get the best out of components.
Pushing boundaries. Think big. Think outside the box. Bold thinking generates results that others did not think were possible.
Questioning assumptions. Achieving new or different results means thinking from first principles, while also doing things which a system was not intended or designed for.
Rigid experimentation with spontaneous thinking. This paradoxical thinking allows modifiers to try lots of hypotheses, while being creative enough to find new solutions.
The modifier is a critical role for building new systems and finding ways to generate consistent results from uncovered insights. They work to uncover secrets and understand what's possible. Modifiers are responsible for exploring possibilities and then exploiting what they've found to make components achieve consistent and build better systems.