LukeW

In his Is Atomic Design Dead? presentation at Smashing Conf New York, Brad Frost discussed the history of design systems and today's situation especially in light of very capable AI models than can generate code and designs. Here's my notes on his talk.

• Websites started as HTML and CSS. People began to design websites in Photoshop and as the number of Web sites and apps increased, the need for managing a brand and style across multiple platforms became clear. To manage this people turned to frameworks and component libraries which resulted in more frameworks and tools that eventually got integrated into design tools like Figma. It's been an ongoing expansion...
• There's been lots of change over the years but at the highest level, we have design systems and products that use them to enforce brand, consistency, accessibility, and more.
• Compliance to design systems pushes from one side and product needs push from the other. There needs to be a balance but currently the gap between the two is growing. A good balance is achieved through a virtuous cycle between product and systems.
• The atomic design system tried to intentionally define use of atoms, molecules, organisms, templates, and pages to bridge the gap between the end state of a product and a design system.
• As an industry, we went too far in resourcing design systems and making them a standalone thing within a company. They've been isolated.
• Design system makers can't be insular. They need to reach out to product teams and work with them. They need to be helping product teams achieve their goals.
• What if there were one global design system with common reusable components? Isn't that what HTML is for? Yes, but it's insufficient because we're still rebuilding date pickers everywhere.
• Open UI tracks popular design systems and what's in them. It's a start to seeing what global component needs for the Web could look like.
• Many pattern libraries ship with an aesthetic and people need to tweak it. A global design system should be very vanilla so you can style it as much as you want.
• The Web still has an amazing scale of communication and collaboration. We need to rekindle the ideas of the early Web. We need to share and build together to get to a common freely usable design system.
• AI models can help facilitate design system work. Today they do an OK job but in the future, fine-tuned models may create custom components on the fly. They can also translate between one design system and another or translate across programming languages.
• This methodology could help companies translate existing and legacy code to new modern design systems. Likewise sketches or mockups could be quickly translated directly to design system components thereby speeding up processes.
• Combining design system specifications with large language models allows you to steer AI generations more directly toward the right kind of code and components.
• When product experiences are more dynamic (can be built on the fly), can we adapt them to individual preferences and needs? Like custom styles or interactions.
• AI is now part of our design system toolkit and design systems are part of our AI toolkit.
• But the rapid onset of AI also raises higher level questions about what designers and developers should be doing in the future? We're more than rectangle creators. We think and feel which differentiates us from just production level tasks. Use your brains, your intuition, and whole self to solve real problems.

In her How to Use AI to Build Accessible Products presentation at Smashing Conf New York, Carie Fisher discussed using AI coding tools to test and suggest fixes for accessibility issues in Web pages. Here's my notes on her talk.

• AI is everywhere. You can use it to write content, code, create images, and more. It impacts how everyone will work.
• But ultimately, AI is just a tool but it might not always be the right one. We need to find the tasks where it has the potential to add value.
• Over 1 billion people on the planet identify as having a disability. Accessible code allows them to access digital experiences and helps companies be complaint with emerging laws requiring accessible Web pages and apps. Businesses also get SEO, brand, and more benefits from accessible code.
• AI tools like Github Copilot can find accessibility issues in seconds consistently, especially compared to the manual checks currently being done by humans. AI can also spot patterns across a codebase and suggest solutions.
• Existing AI coding tools like Github Copilot are already better than Linters for finding accessibility issues.
• AI can suggest and implement code fixes for accessibility issues. It can also be added to CI/CD pipelines to check for accessibility issues at the point of each commit. AI can also serve as an accessibility mentor for developers by providing real-time suggestions.
• More complex accessibility issues especially those that need user context may go unfound when just using AI. Sometimes AI output can be incomplete or hallucinate solutions that are not correct. As a result, we can't over rely on just AI to solve all accessibility problems. We still need human review today.
• To improve AI accessibility, provide expanded prompts that reference or include specifications. Code reviews can double check accessibility suggestions from AI-based systems. Regularly test and refine your AI-based solutions to improve outcomes.
• Combing AI and human processes and values can help build a culture of accessibility.

Since launching the Ask Luke feature on this site last year, we've added the ability for the system to respond to questions about product design by citing articles, videos, audio, and PDFs. Now we're introducing the ability to cite the thousands of images I've created over the years and reference them directly in answers.

Significant improvements in AI vision models have given us the ability to quickly and easily describe visual content. I recently outlined how we used this capability to index the content of PDF pages in more depth making individual PDF pages a much better source of content in the Ask Luke corpus.

We applied the same process and pipeline to the thousands of images I've created for articles and presentations over the years. Essentially, each image on my Website gets parsed by a vision model and we add the resulting text-based description to the set of content we can use to answer people's design questions. Here's an example of the kinds of descriptions we're creating. As you can see, the descriptions can get pretty detailed when needed.

If someone asks a question where an image is a key part of the answer, our replies not only return streaming text and citations but inline images as well. In this question asking about Amazon's design changes over the years, multiple images are included directly in the response.

Not only are images displayed where relevant, the answer refers to them and often refers to the contents of the image. In the same Amazon navigation example, the answer refers to the green and white color scheme of the image in addition to its contents.

Now that we've got citations and images steaming inline in Ask Luke responses, perhaps adding inline videos and audio files queued to relevant timestamps might be next? We're already integrating those in the conversational UI so why not... AI is a hell of a drug.

Additional articles about what I've tried and learned by rethinking the design and development of my Website using large-scale AI models.

• New Ways into Web Content: rethinking how to design software with AI
• Integrated Audio Experiences & Memory: enabling specific content experiences
• Expanding Conversational User Interfaces: extending chat user interfaces
• Integrated Video Experiences: adding video experiences to conversational UI
• Integrated PDF Experiences: unique considerations when adding PDF experiences
• Dynamic Preview Cards: improving how generated answers are shared
• Text Generation Differences: testing the impact of AI new models
• PDF Parsing with Vision Models: using AI vision models to extract PDF contents
• Streaming Citations: citing relevant articles, videos, PDFs, etc. in real-time
• Streaming Inline Images: indexing & displaying relevant images in answers

Big thanks to Sidharth Lakshmanan and Sam Breed for the development help.

New technology companies often have grand ambitions. And for good reasons - ambitious plans help recruit talent, raise capital, and set the bar high. But progress toward these high-level goals relies on identifying and excelling at much lower-level use cases.

It's very common for new technology companies to aspire being "the platform for... the Internet of things, AI analytics, mobile testing, etc." Being a platform means you capture a lot of uses cases or to put it more simply... people use your service for a lot of different things. And more use equals more value.

But vision is not the same as strategy. Vision is about the end goal. It paints a picture of the future state you're aiming for. It’s what you want to achieve. Strategy, on the other hand, is how you get there.

When you use a broad vision as a strategy, you end up having a hard time making decisions and rationalizing a never-ending set of opinions. With a strategy like “we’ll be the platform for the Internet of things”, everyone has an opinion on how things on the Internet should work -which one do we listen to?

Consider instead a specific market for the Internet of things, like home automation, and an even more specific use case for home automation like "controlling the temperature in your house". It's much easier to evaluate decisions about what a good experience for controlling the temperature in your house is than for "we’ll be the platform for the Internet of things”.

But if you focus on such a narrow use case, how will you ever build a big business? I'm not suggesting abandoning the big vision instead I'm advocating for having a strategy based on solving concrete uses cases to get there. Let's look at another example: Yelp.

Today, Yelp is used for recommendations for all kinds of services: skydiving training, auto body shops, tea parlors, and more. But it didn't start that way. Yes, Yelp likely started with the ambitious vision of being a platform for all service recommendations. But it first launched in San Francisco with restaurant reviews. A very specific market and very specific use case.

Why start with restaurants? A good starting use case is the one with the most acute pain. In the context of services, people need to eat three times a day. They get their hair cut once a month and maybe need a plumber once a year. So where should Yelp start? Probably restaurants.

When solving for a specific use case, it's important to build with the bigger vision in mind and not paint yourself into a corner of only being useful for one thing. But you definitely have to be great at solving each use case your platform supports. How else will you convince people to adopt your solution? Once you can demonstrate clear value for a specific use case, you can tackle more (likely adjacent ones).

This way of scaling ensures your solution is actually good at addressing a concrete problem people have not just an abstract vision. When you hear "What's your platform for? Well... you can use it for pretty much anything." in a sales pitch, that's a warning sign.

When you instead address specific use cases well, you learn what parts of your platform matter the most by identifying patterns and doubling down on them. It's only from solving highly specific use cases that you actually get to a platform that can be broadly used for many different things. And why Amazon started by only selling books on the Web.

The Ask Luke feature on this site uses the thousands of articles, hundreds of PDFs, dozens of videos, and more I've created over the years to answer people's questions about digital product design. Since it launched a year ago, we've been iterating on the core of the Ask Luke system: retrieving relevant content to improve answers.

The most important job of any product interface is making its value clear and accessible to people. Most apps resort to some form of onboarding to accomplish this, but it's exponentially more impactful to experience value than to be told it exists. Likewise it's much more effective to learn through using an interface than through a tutorial explaining it.

These two factors make the seemingly simple job of "getting people to product value" quite difficult. Compounding the issue is that fact that interface solutions that accomplish this often feel simple and obvious -but only after they're uncovered. So iterating to an interface that intuitively conveys value and purpose is usually an iterative process.

That's a long-winded introduction, but it's important context for the changes we made to Ask Luke. The purpose and value of this feature is to pull the most relevant bits of my writings, videos, audio, and files together to answer people's questions about digital product design. So we made a bunch of changes to make that even more front and center -to make how Ask Luke works more obvious.

Now as answers to people's questions stream in, we add citations to the relevant articles, videos, PDF, etc. being used to answer a question in real-time. We also add these citations to the list of sources on the right dynamically instead of all at once before a question is answered.

Before people were able to select any given source and view it in the Ask Luke conversational UI. With these updates, they are also taken to the relevant part of a source: to the relevant point in a video; to the relevant page in a PDF. Since this is easier to see than read about, here's a quick video demonstrating these changes and hopefully making the value and purpose of Ask Luke a bit more obvious.

• Integrated Audio Experiences & Memory: enabling specific content experiences within a conversational UI
• Expanding Conversational User Interfaces: extending chat user interfaces to better support AI capabilities
• Integrated Video Experiences: adding video-specific experiences within conversational UI
• Integrated PDF Experiences: unique considerations when adding PDF experiences
• Dynamic Preview Cards: created on the fly to improve sharing answers
• Text Generation Differences: testing the impact of AI new models
• PDF Parsing with Vision Models: using AI vision models to extract content form PDFs

Big thanks to Sidharth Lakshmanan and Sam Breed for the engineering lift on these changes.