When California Gov. Gavin Newsom vetoed a key piece of AI oversight legislation Sunday, he said he did so because the measure “falls short of providing a flexible, comprehensive solution to curbing the potential catastrophic risks.”
He then said he “has asked the world’s leading experts on genAI to help California develop workable guardrails for deploying genAI, focusing on developing an empirical, science-based trajectory analysis of frontier models and their capabilities and attendant risks.”
Those would be laudable sentiments if any of them had any chance of actually delivering a more secure and trustworthy environment for Californians. But Newsom, one of the nation’s smarter politicians, well knows that such an effort is a fool’s mission. I could add cynically that the governor merely wants to be seen trying to do something, but why state the obvious?
Problem One: GenAI deployments are already happening and the technology is being deeply embedded into untold number of business operations. It’s all-but-ubiquitous on the major cloud environments, so even an enterprise that has wisely opted to hold off its genAI efforts for now would still be deeply exposed. (Fear not: There are no such wise enterprises.)
The calendar simply doesn’t make sense. First, Newsom’s experts get together and come up with a proposal, which in California will take a long time. Then that proposal goes to the legislature, which means lobbyists will take turns watering it down. What are the chances the final result will be worthy of signature? Even if it is, it arrive far too late to do any good.
Candidly, given how far genAI has progressed in the last two years, there’s a fine chance that had Newsom signed the bill into law on Sunday, it would have still been too late.
Part of the reason for that is because the enforcement focus is on AI vendors, and it is highly unlikely that state regulators will be able to effectively perform oversight on something as complex as genAI development is today.
In his veto message, Newsom pointed to the flaw of vendor oversight, but zeroed in on the wrong reason.
“By focusing only on the most expensive and large-scale models, SB 1047 establishes a regulatory framework that could give the public a false sense of security about controlling this fast-moving technology,” he said. “Smaller, specialized models may emerge as equally or even more dangerous than the models targeted by SB 1047 — at the potential expense of curtailing the very innovation that fuels advancement in favor of the public good.”
In short, the governor is arguing that regulators shouldn’t only look at the biggest players, but focus on many of the smaller specialty shops as well. That argument makes sense in a vacuum. But in the real world, regulators are understaffed and under-resourced to effectively manage a handful of major players, much less the many niche offerings that exist. It sounds great spoken from a podium, but it’s not realistic.
Here’s the real problem: No one in the industry — big players included — truly knows what genAI can and can’t do. No one can accurately predict its future. (I’m not even talking about five years from now; experts struggle to predict capabilities and problems five months from now.
We’ve all seen the dire predictions of what might happen with genAI. Some are overblown — remember the extinction reports from February? And some are frighteningly plausible, such as this Cornell University report on how AI training AI could lead to a self-destructive loop. (By the way, kudos to Cornell’s people for comparing it to Mad Cow disease. But to make the analogy work, they created the term Model Autophagy Disorder so they could use the acronym MAD. Sigh.)
There is a better way. Regulators — state, federal and industry-specific — should focus on rules for enterprises and hyperscalers deploying genAI tools rather than the vendors creating and selling the technology. (Granted, the big hyperscalers are also selling their own flavors of genAI, but they are different business units with different bosses.)
Why? First of all, enterprises are more likely to cooperate, making compliance more likely to succeed. Secondly, if regulators want vendors to take cybersecurity and privacy issues seriously, take the fight to their largest customers. If the customers start insisting on the government’s rules, vendors are more likely to fall in line.
In other words, the paltry fines and penalties regulators can threaten are nothing compared to the revenue their customers provide. Influence the customers and the vendors will get the message.
What kind of requirements? Let’s consider California. Should the CIO for every healthcare concern insist on extensive testing before any hospital uses genAI code? Shouldn’t those institutions face major penalties if private healthcare data leaks because someone trusted Google’s or OpenAI’s code without doing meaningful due diligence? What about a system that hurts patients by malfunctioning? That CIO had better be prepared to detail every level of pre-launch testing.
How about utilities? Financial firms? If the state wants to force businesses to be cautious, there are ways of doing so.
Far too many enterprises today are feeling pressured by hype and being forced by their boards to jump into the deep end of the genAI pool. CIOs — and certainly CISOs — are not comfortable with this, but they have nothing to fight back with. Why not give CIOs a tool with which to push back: state law.
Give every CEO an out for not risking their businesses and customers by accepting magical-sounding predictions of eventual ROI and other benefits. Regulators could become CIOs’ new best friends by giving them cover to do what they want to do anyway: take everything slowly and carefully.
Trying to regulate vendors won’t work. But giving political cover to their customers? That, at least, has a real chance of succeeding.
Whatever Apple’s long-term plans for satellite connectivity, one facet that cannot be ignored is that its Messages via Satellite system is already saving lives — including among iPhone users affected last week by the horror of Hurricane Helene.
The aftermath of Helene
While there seems no end to all the bad news playing out worldwide at this time, a sizable chunk of the United States was particularly impacted by Helene. When that storm hit, it wiped out power grids and cell service, wrecked infrastructure and took scores of lives across several US states. The hurricane wiped out communications for inland, leaving victims stranded with no way of getting help (particularly in light of a massive Verizon outage at the same time).
That is, unless people had iPhones. Reports have appeared on social media explaining how compatible iPhones running iOS 18 enabled those impacted by the storm to send and receive messages via satellite to seek help or let family know they were safe. Apple is well aware of the damage wrought by this disaster; company CEO Tim Cook has promised the company will donate to support relief efforts on the ground.
What is Messages via Satellite?
Available in the US with iOS 18 on iPhone 14s or later models, Messages via Satellite allows users to send and receive texts and other communications using iMessage and SMS when a cellular or Wi-Fi connection is not available.
“Messages via Satellite automatically prompts users to connect to their nearest satellite right from the Messages app to send and receive texts, emoji, and Tap backs over iMessage and SMS,” Apple explained. “Because iMessage was built to protect user privacy, iMessages sent via satellite are end-to-end encrypted.”
Messages via Satellite is essentially an extension to the SOS via satellite service Apple introduced in 2022. It’s available at present only in the US and Canada.
How it works
To receive messages, you or your contact must be running iOS 18, iPadOS 18, macOS Sequoia, watchOS 11, visionOS 2, or later.
To use Messages via Satellite, follow these steps:
First, you must be outside with a clear view of the sky and horizon.
Open Messages, and if you have no cellular or Wi-Fi coverage, a prompt appears.
Tap Use Messages via Satellite.
Follow the instructions to connect to a satellite.
You will then need to select Messages from the selection of services that appear.
Enter your message and tap send.
The message is likely to take longer than usual to send.
Contacts receiving your message will see a status message to show you’re using satellite.
You can also use SMS via satellite — just open Settings>Apps>Message and turn on Send as Text Message and then connect to satellite to send. To reply to SMS messages via satellite, requires iOS 17.6 or later.
When will the service be international?
Apple’s partner in satellite connectivity, Globalstar, continues to launch new satellites to support the expanding service. Regulatory filings from that company suggest it hopes to launch an additional 26 satellites by next year, with at least one report claiming it will have 3,000 in place eventually. At least one space expert thinks Apple will eventually choose to widen the network to become a full satellite-based communication service.
It is likely Apple will follow a cadence similar to the manner in which it made Emergency SOS via satellite available once that service was initially launched in the US and Canada. It opened up in France, Germany, Ireland, the UK, Australia, Austria, Belgium, Italy, Luxembourg, New Zealand, Portugal, Switzerland, Spain, and the Netherlands across the following year and in Japan a year later.
A lesson for everyone
All of this is important in terms of saving lives and providing reassurance for families and friends of those in the disaster-hit areas, but the fact that these devices have helped maintain community resilience amid disaster might also be a salutatory lesson in business resilience. After all, other than avoiding platforms characterized by frequent ransomware attacks and spiralling ancillary security support costs, it just might be that smartphones equipped with satellite connectivity could become a vital business asset as we navigate an increasingly uncertain world.
After all, why should SpaceX dominate such an economically and socially essential asset as satellite communications? It makes sense for every business to ensure there are multiple providers of such a strategic essential — particularly to maintain business and community resilience.
The European Commission has appointed a group of AI specialists to outline how businesses should comply with forthcoming AI regulations.
The group includes prominent figures like AI pioneer Yoshua Bengio, former UK government adviser Nitarshan Rajkumar, and Stanford University fellow Marietje Schaake.
Microsoft’s ambitious collaboration app, Microsoft Loop, includes shared workspaces as well as portable content snippets called Loop components. These components can be shared and embedded in multiple Microsoft 365 apps.
What makes Loop so useful is that those shared components can be updated by multiple collaborators, and the contents of these components stay in sync no matter where they’re embedded. One person could edit a component in an Outlook email, while another edits it in a Teams chat, and the latest changes appear in both places.
We have a separate guide that covers Microsoft Loop more broadly and details how to use the Loop app itself. But you don’t actually need the app to use Loop components. That’s because Loop components can be integrated into several Microsoft 365 apps, so you can create, share, and work on them in an app you’re already familiar with. That’s what we’ll cover in this guide.
In this article
What is a Microsoft Loop component?
What apps can I use Loop components in?
Who can use Loop components in Microsoft 365 apps?
Creating a Loop component
Sharing your Loop component
Interacting with a Loop component
Managing your Loop components
What is a Microsoft Loop component?
A Loop component is a portable text card or content snippet — in list, paragraph, table, or another format — that you and your co-workers can collaborate on synchronously or asynchronously.
For example, if you create a Loop component that contains a table, you and your collaborators can add, change, or remove numbers or text, or adjust the table’s formatting. When someone makes a change to the table, you and your co-workers can see it happen, and see who’s doing the change, in real time.
Loop components can be embedded in (and are cross-compatible among) a subset of Microsoft 365 apps including Outlook, Teams, and Word. When you create a Loop component in one of these M365 apps, you can copy and paste the link to it into another M365 app — and will then be able to continue working on the component in that app.
Imagine that you create a Loop component with a task list on it in a Teams chat. After doing this, you copy and paste a link to it into an Outlook email. Any changes that you or others make to the task list in the Teams chat will automatically appear in the email — and the recipient of your email can also make changes to the task list that will appear in the Teams chat.
What apps can I use Loop components in?
The five main Microsoft 365 apps that Loop components can be used in today are OneNote, Outlook, Teams, Whiteboard, and Word, with some limitations:
OneNote: Loop components are gradually rolling out to the OneNote Windows and web apps but are not yet available in the macOS or mobile apps.
Outlook: Loop components are available in the Windows and web apps but not in the macOS or mobile apps.
Teams: Loop components are available in the Teams Windows, macOS, Android, iOS, and web apps.
Whiteboard: Loop components are available in the Whiteboard Windows, web, Android, and iOS apps. In the mobile apps, you can currently only view and edit Loop components; component creation and copy/paste functions will be added in the future. People collaborating with you on a whiteboard who are not your team members are unable to view, edit, create, or copy and paste Loop components.
Word: Loop components are available in the web version of Word, but not in the desktop or mobile apps.
Who can use Loop components in Microsoft 365 apps?
Only users who have Microsoft 365 business, enterprise, or education accounts can embed Loop components in Microsoft 365 apps, and only other users within your organization can use them within M365 apps.
That said, anyone with a free or paid Microsoft 365 account can create and use Loop workspaces, pages, and components in the Loop app, as covered in our Microsoft Loop cheat sheet. Because Loop components are the same whether they’re embedded in a Loop page or a Microsoft 365 app, it’s worth your while to keep reading this story to learn about the various types of Loop components and the elements you can include in them.
Creating a Loop component
Here’s how to start a Loop component in each of the five apps.
In Outlook: You can insert a Loop component inside an email message. If the recipient is in your Microsoft 365 organization or has a Microsoft user account, they can interact with the Loop component when they open your email.
In the toolbar above the email that you’re composing, click the Loop icon. A panel will open listing the Loop components that you can select to insert. (We’ll go over the Loop component types in the next section of the story.)
Click the Loop icon in Outlook, then choose one of the Loop component types.
Howard Wen
In Teams: You can insert a Loop component inside a Teams chat or in a post in a Teams channel. You and others in the chat or channel will be able to collaborate on the component.
On the toolbar for your message, click the Loop icon: it’s at the bottom right for a chat conversation and at the bottom left for a channel post. A Loop component composition window will open in the channel or chat thread. Buttons to insert specific components will appear along the bottom of this window — click the three-dot icon at the bottom right corner to see more selections.
Inserting a Loop component in a Teams chat.
Howard Wen
In OneNote or Word: Set the cursor where you want to embed a Loop component in your document or page. On the toolbar across the top, select Insert and then Loop Component (or Loop Components). A panel will open listing the Loop components that you can select to insert.
Adding a Loop component to a OneNote page.
Howard Wen
In Whiteboard: Open a whiteboard. Click the three-dot icon on the bottom toolbar. Select Loop components from the small panel that opens.
Inserting a Loop component in a whiteboard.
Howard Wen
After you’ve selected a Loop component (see below for the main types available), a draft of the component appears in the Microsoft 365 app you’re using. Click Add a title and type in a title for your new Loop component.
The Loop component types
Below are the main Loop components that you can insert into the Microsoft 365 apps. Over time, Microsoft may add more components.
Lists: You can insert a list component in bulleted, numbered, or checklist format. To the right of a new bullet point or number in those list types, type in text for the first item on your list and press the Enter key. A second bullet/number will appear below the first, and you can type in the words for your second item. Repeat until you’ve entered all items for your list.
You set up a checklist the same way, but each item has a circle by it. Clicking the circle will insert a checkmark and cross off its corresponding item to mark it as complete. Clicking the circle again will remove the checkmark and strikethrough.
A checklist component in Teams.
Howard Wen
Paragraph: This inserts a standard text block where you can type words, sentences or multiple paragraphs.
Table: The basic table template has two rows and two columns by default. To insert a new column, move the pointer over a vertical line in the table, then click the plus sign that appears at the top of the line. To insert a new row, move the pointer over the left side of a horizontal line in the table, then click the plus sign that appears.
Creating a table component in Outlook.
Howard Wen
To fill in a table, click inside each empty cell, then type to fill it in. To change a column header, move your pointer over it, click the down arrow that appears at its right, select Rename, and type in a new name.
Task list: This is technically a table template with preset headers. Fill in the task names, the names of co-workers you want to assign each item to, and the due dates. When a task is complete, click the circle next to it.
Q&A: This is a list on which you and your co-workers can post questions and answer each another’s questions. Click Ask a new question and type in your question. To reply to a question, click Answer below it and type in your answer.
A Q&A component in an Outlook email.
Howard Wen
Voting table: This is another table template. It helps you present ideas that your co-workers can vote on.
Progress tracker: This table template helps you track projects that you and your co-workers are collaborating on.
Kanban board and Team retrospective: These are similar templates that help you set up your projects as a series of color-designated cards. They feature the same easy-to-use, robust interface.
The kanban board template lets you add cards for various tasks, then move them under different headers indicating their status as they progress.
Howard Wen
Code: In OneNote and Teams you may also see the option to insert a code block, useful for developer collaboration.
Tip: Inside many areas of a Loop component, you can tag a co-worker who’s in your Microsoft 365 organization by typing @ followed by their name. You do this to bring their attention to your Loop component if you want them to view it or collaborate with you on it. They will get a notification through email or Teams.
Adding other elements to your Loop component
If you click the space toward the bottom of your Loop component, the words “Just start typing…” appear. You can type text inside this space if you want to provide more information to append to your Loop component.
Or, if you press the / key, a menu will open that lists several elements that you can add below your Loop component. For example, you can append an additional table or list. But there are other, unique elements that you may find useful:
Date: When you select this, a mini-calendar will open. Click a date on it, and it’ll be inserted as a line of text in your Loop component.
Callout: Select this and type in text that you want to be set off with a lightly shaded background. The callout will also be denoted with a pushpin icon; you can change this icon by clicking it, and on the panel that opens, selecting another icon.
The Callout element has a shaded background and a pushpin icon to make it stand out.
Howard Wen
Table of contents: This is a really useful element when you’re working on a Word document. Select this and a table of contents will be generated based on the paragraphs and section headings of the document.
Divider: If you add several elements, insert divider lines between them to make your Loop component look better organized and less confusing.
Headings: You can insert a bold text heading, choosing from three sizes. Or you can insert a collapsible heading: the first line is the heading, and the second and any subsequent ones are regular formatted text. When you click the arrow to the left of the heading, this will “collapse” the lines of regular text, folding them up into the heading. Clicking this arrow again will reveal them again.
Quote: This is simply text that you want to have set off within your Loop component, bringing more attention to it.
Person: This is another way to tag a co-worker. You can select this instead of typing @.
When you tag a co-worker in a component, they’ll receive a notification.
Howard Wen
Emoji picker: Obviously, this is for inserting an emoji somewhere in your component. Selecting this will pull up a panel filled with lots of emoji that you can scroll through.
Label: You can select from preset labels (such as Not started, In progress, Completed, etc.) to insert and optionally type in a few words of explanation. To create a set of custom labels, select Add label group, then type in a name for the new group along with the individual label options.
The label in a component can be changed later (e.g., from In progress to Completed) by clicking it and selecting another option from the Label panel.
Image: You can insert an image file that’s stored on your PC’s drive or in OneDrive.
As you become more familiar with these elements, you can skip scrolling through the list of elements by typing / followed by the first letter or two of the element you want. To insert an image, for example, type /i and select Image.
A Loop component with multiple elements.
Howard Wen
Note that many of these elements can be combined. For example, you can insert a date, emoji, image, or person element inside a table cell. And some elements can be inserted alongside one another, sharing the same line. Go ahead and play around to see which combinations work.
As you add several elements, you can move any of them to a higher or lower spot within the component. Click to select the element, then click-and-hold the six-dot icon to the left of the element. Drag this icon up or down, and then let go where you want the element to be moved.
Sharing your Loop component
Once you’ve assembled your Loop component, you’re almost ready to send it to your co-workers for collaboration. But first, think about who you want to share it with.
Changing your Loop component’s share settings
By default, Loop components are accessible (and editable) by anyone in your organization, but you can change that.
In Outlook: Along the upper left of your Loop component, click your Loop component’s name. (It’ll either be derived from the subject line of your new email or named “Loop component [number]”) On the small panel that opens, select People in [your organization] with this link can edit.
In Teams: Along the top of your Loop component, click People in your organization with the link can edit.
The “Link settings” panel opens.
Changing a component’s sharing settings in Teams.
Howard Wen
Below “Share the link with,” you can select:
Anyone
People in [your organization]
Recipients of this message (if the component is in an Outlook email)
People currently in this chat (if the component is in a Teams chat)
Only [channel name] (if the component is in a Teams channel)
People with existing access
Note: Your organization may have disabled one or more of these options and/or set up different default sharing permissions.
If you’d prefer that other people you share with not be able to make changes to your Loop component, below “More settings,” click Can edit and change it to Can view.
Additionally, you can set an expiration date. On this date, the component will no longer be viewable by the people you’ve shared it with. (This feature is currently available only in Teams.)
In OneNote, Whiteboard, and Word: Components embedded in these three apps use the same share settings that you set up for the entire notebook, whiteboard, or document.
To share a OneNote notebook or Word doc, click the Share button at the upper right of the page. Select Manage Access and on the panel that opens, select Start sharing. In Whiteboard, simply click the Share button at the upper right of the page.
On the panel that opens for any of these apps, type in the names, groups, or emails for people that you want to share the notebook, whiteboard, or document with. To change access permissions, click the Can edit (pencil) icon and change it to Can view.
Sending your Loop component
After you’ve finished setting up your Loop component and its access permissions, you’re ready to share it with your co-workers.
In Outlook: Fill out any other areas in the email body before or after your Loop component. When you’re finished composing your email, click the Send button.
In Teams: Click the arrow button at the lower right. Your Loop component will be inserted into your Teams conversation.
In OneNote, Whiteboard, and Word: Once you’ve shared the notebook, whiteboard, or document as described above, your co-workers will get a notification through email.
Resharing your Loop component in other M365 apps
You can copy your Loop component and embed it into other Microsoft 365 apps. Click the Copy component icon (two overlapping rectangles) at the component’s upper right. This will copy a link to it to your PC clipboard.
Click the Copy component icon to copy a link to it.
Howard Wen
Here’s what happens when you paste this link in another app or location:
When you paste this link inside a Microsoft 365 app that supports Loop, your Loop component will appear inside that app. So if you create a Loop component in a Teams chat, you can paste it inside a different Teams chat or channel, into a new Outlook email, or into a page in OneNote, Whiteboard, or Word. Your co-workers will be able to contribute to your Loop component in the other app or location.
When you paste the link into an app that doesn’t support Loop, a link to open the component in a browser will appear. Your co-workers will still be able to collaborate on the component, but not directly in the app where you pasted the link.
Interacting with a Loop component
The entire point of a Loop component is for you and your co-workers to collaborate on it. If multiple collaborators are looking at the component at the same time, everyone can see changes happen in real time and who’s making the changes. If someone looks at the component later, they’ll see all changes made earlier.
To change items in a Loop component: Click on the text or other element (date, image, table, etc.) you want to change and make your change.
To add an element to a Loop component: Click the space toward the bottom of the Loop component. The words “Just start typing…” appear. You can type in text or press the / key to see the same list of options covered under “Adding other elements to your Loop component” above.
To add a comment to an element: You and your co-workers can add comments to most elements. Click the element to select it, then click the icon of two speech balloons at the lower left of it. On the panel that opens, click Comment and on the card that opens, type a brief comment and optionally select an emoji.
You and your collaborators can add comments to most elements in a Loop component.
Howard Wen
You can access these additional functions along the top of the Loop component:
To view a Loop component inside a browser: At the upper left of your Loop component, click its name. Your Loop component will open in the Loop app in a new tab in your browser. You can make changes to the Loop component in this browser tab.
It’s easy to rename a component in the Loop app.
Howard Wen
To rename the component, click its name in the title bar at the top of the page. This opens a bar that lists the file location of this component — click the name of the component at the end of this bar to rename the component.
To see where a Loop component is being shared: Click the cloverleaf (“Shared locations”) icon at the upper right of the component to see the apps that your Loop component is being shared in.
To add a Loop component to a Loop workspace: Click the cloverleaf (“Shared locations”) icon at the upper right of the component. On the panel that appears, select Add to Loop workspace and select a workspace to add it to.
To copy (a link to) a Loop component: As noted above, you can embed a Loop component you’ve created in various Microsoft 365 apps. You can also embed a component created by someone else who granted you permission to edit it. Click the dual-rectangle icon to copy a link to it to your PC clipboard, then paste it into another app. (See “Resharing your Loop component in other Microsoft 365 apps” above for details about how this works.)
To see who has access to a Loop component: Click the dual silhouette icon.
You can quickly see who has access to a Loop component.
Howard Wen
To change the sharing status of a Loop component (in Teams): Move the pointer over your Loop component and click the pencil icon that appears at the upper right of it. Then along the top of your Loop component, click People in your organization with the link…, then follow the instructions above under “Changing your Loop component’s share settings.”
To delete a Loop component: Move the pointer over the Loop component until a toolbar with emojis appears at the upper right of it. Click the three-dot icon, and on the menu that opens, select Delete.
Note: If you created the Loop component, you can delete it. If you reshare a Loop component that someone else created, you can only delete it from the app that you reshared it on.
To pin a Loop component in Teams: Move the pointer over the Loop component until the toolbar with emojis appears at the upper right. Click the three-dot icon, and on the menu that opens, select Pin.
Pinning a Loop component in a Teams chat.
Howard Wen
If you’re in a Teams chat, this will place a horizontal bar with the name of your Loop component along the top of the chat window. Now, no matter how far down the stream of messages or chats has progressed, clicking this bar will jump your view back up to your Loop component.
Pinning a component in a Teams channel is less useful. Instead of pinning a shortcut to the component at the top of the page, it simply places a pushpin icon on the component. It is easier to see that way, but you still have to scroll through the list of posts.
Managing your Loop components
Most of the Loop components you’ve created from within a Microsoft 365 app are stored in your OneDrive and count toward whatever storage limit comes with your Microsoft 365 plan. You’ll find them under “My files” in different folders depending on the app you created them in:
Components created in OneNote are in the OneNote Loop Files folder.
Components created in Outlook are in the Attachments folder.
Components created in Teams chats are in the Microsoft Teams Chat Files folder.
Components created in Whiteboard are stored in the Whiteboards > Components subfolder.
Components created in Word are stored in the Word Loop Files folder.
Loop components that you create in M365 apps are stored in your OneDrive.
Howard Wen
(Loop components you create in a Teams channel are not stored in your own OneDrive, but in the SharePoint site for the team that houses the channel, under Documents > [channel name].)
In OneDrive, you can manage your Loop components as you would any other file: right-click a component’s file name to see a menu that lets you copy, delete, or rename it; manage its access settings; and more.
This story was originally published in April 2023 and updated in September 2024.
Like many small and medium-sized businesses (SMBs), ‘JPX Construction’ didn’t see itself as a potential target for a ransomware attack.
It wasn’t a high-profile organisation. It didn’t seem to hold much high-value data. So, when the financial director turned on her PC to find a black screen with a ransomware message, it came as a painful shock.
That’s the premise of a dramatic new film that takes viewers inside the events of a fictionalised ransomware attack based on true events.
Cyber threats remain an acute threat: the UK Government’s 2024 Cyber Security Breaches Survey revealed that 47% of micro-sized businesses and 58% of small businesses had experienced some form of breach or attack in the last 12 months. Meanwhile 19% of micro businesses and 29% of small businesses fell victim to a cybercrime.
Butterfly, a new film by Dark Matter in association with McAfee and Dell Technologies, explores this experience in-depth.
The film centres on an attack on an SMB in the construction industry, following the events from the perspective of three key personnel: the financial director, the external IT consultant and the company’s Managing Director.
In it, Finance Director Georgia soon discovers that it’s not just her PC that’s affected, but PCs across the firm and its servers too.
She calls in the company’s external IT consultant, Noah, in the hope of fixing the problem. But with the system drives and servers encrypted, and all day-to-day business stalled, Noah has no choice but to deal with the attackers directly.
The film makes clear that there are no easy answers, even with the police and cybersecurity experts involved.
The choice of whether to pay or not to pay the ransom is agonizing, with strong arguments on both sides. Is it worth paying up to minimise impact on the business, or do you make yourself a target for repeat attacks?
Organised crime
As Georgia, Noah and MD Stephen weigh their options and negotiate with their attackers, it also becomes clear that they’re not dealing with individual hackers or opportunists, but a structured and well-organised criminal group.
Cyber crime has evolved into a commercial enterprise, and JPX Construction finds itself the unwilling ‘customer’ of a faceless organisation that it cannot trust.
Further complications only increase the ransom demands, and the financial impact grows from tens to hundreds of thousands of pounds, with the ongoing effects – including loss of business and reputational damage – lasting months.
What’s more, this comes with profound physical and emotional effects that might surprise you.
As Georgia notes on screen, the biggest issue raised is that so few SMBs recognise that ransomware could happen to them or implement the basic steps that could ward off an attack.
As the Cyber Security Breaches Survey 2024 reveals, only 14% of microbusinesses and 30% of small businesses have had cybersecurity training or awareness-raising sessions in the last 12 months.
The belief that you’re not a target is a common misconception, says Jake North, Product Manager for Consumer and Small Business Security at Dell Technologies.
“Small businesses undervalue what they do” he says, “and they are also in a fairly unprotected state, because they don’t have the IT resources to manage it.”
Butterfly contains a wealth of information and insight on what it’s like to be at the sharp end of a ransomware attack.
To view Butterfly: A True Cybercrime Story, sponsored by McAfee and Dell Technologies, click here
The CHIPS and Science Act provides $52.7 billion for American semiconductor research, development, manufacturing, and workforce development. The Act’ provides $39 billion in manufacturing incentives, including $2 billion for the legacy chips used in automobiles and defense systems; $13.2 billion in R&D and workforce development; and $500 million for international information communications technology security and semiconductor supply chain activities.
That R&D money includes $11 billion in funding to advance four programs: the National Semiconductor Technology Center (NSTC); the National Advanced Packaging Manufacturing Program (NAPMP); the CHIPS Metrology Program; and the CHIPS Manufacturing USA Institute. The Act also provides a 25% investment tax credit for capital expenses for manufacturing of semiconductors and related equipment.
The CHIPS Act’s purpose was to strengthen American supply chain resilience after problems caused by the COVID-19 pandemic and to counter China’s rising share of the market. The US share of global semiconductor fabrication capacity has fallen from about 36% in 1990 to about 10% in 2020, according to a Congressional Research Service report. Meanwhile, China’s share of chip manufacturing has grown nearly 50% over the past two years and now comprises about 18% of the world’s supply.
In 2023, the Department of Commerce, which is administering the CHIPS Act, spent months negotiating with semiconductor designers and fabricators to gain commitments from the companies and to achieve specific milestones in their projects before getting government payouts. For example, negotiations between the federal government and TSMC resulted in the Taiwanese semiconductor designer and manufacturer being promised $6.6 billion in CHIPS Act funding; in return, the company pledged to bring its most advanced 2nm process technology to US shores and added plans for a third fabrication plant to its Arizona site.
The White House has argued that CHIPS Act spending will grow America’s share of the world’s leading-edge chip market to 20% by 2030. But industry experts say more government incentives will be needed to sustain and continue that growth domestically.
According to Mario Morales, a group vice president at IDC, the current CHIPS Act is just the start; he expects the Biden Administration to champion a second CHIPS Act that will spend even more money and likely be approved sometime around 2026 or 2027. There may also be a third CHIPS Act after that, Morales said.
The current CHIPS Act was passed by Congress and signed into law by US President Joseph R. Biden Jr. on Aug. 9, 2022.
Since December 2023, the Commerce Department has allocated about $32 billion in funding among chipmakers, including Samsung, TSMC and Intel. In return, various chip designers and makers have pledged about $300 billion in current and future projects in the US, according to the White House.
Here’s a timeline of where the money is going, with the most recent allocations listed first:
September 2024
Intel is set to receive $8.5 billion CHIPS Act funding. The deal, which has been in negotiation for months, would represent the largest subsidy package awarded to date under the act, which aims to reduce US reliance on foreign chip production and strengthen the country’s semiconductor supply chain.
The deal could be finalized by the end of the year, according to one report, but there’s no guarantee of that.
To date, the CHIPS Act has allocated proposed funding across 17 companies, 16 states, and 26 projects. However, no CHIPS funding has actually been disbursed yet. The Department of Commerce expects to begin disbursing awards over the coming weeks and months.
April 2024
Micron, which plans to build two new fabrication plants in upstate New York and another in Boise, Idaho – where its headquarters is located, got $6.14 billion in funding.
Samsung got $6.4 billion to build leading-edge logic, R&D, and advanced packaging fabs in Taylor, TX, and to expand a current-generation and mature-node facility in Austin, TX.
TSMC got $6.6 billion to support the development of three greenfield leading-edge fabs in Phoenix, AZ.
March 2024
Intel was awarded $8.5 billion in funding, the most of any CHIPS Act allocations to date. Intel expects to use the money to advance its commercial semiconductor projects in Arizona, New Mexico, Ohio and Oregon. The company also said the funds would create more than 10,000 company jobs and nearly 20,000 construction jobs, and would support more than 50,000 indirect jobs with suppliers and supporting industries.
February 2024
NSTC in Albany, NY was promised more than $5 billion in funding. The NSTC is a public-private partnership that will perform research on next-generation semiconductor technologies by supporting the design, prototyping, and piloting of the latest semiconductor technologies.
GlobalFoundries, in Malta, N.Y. and Essex Junction, VT. (GF) is expected to receive about $1.5 billion to help them expand and create new manufacturing capacity and capabilities for automotive, IoT, aerospace, defense, and other markets. GF’s chips are used in everything from blind spot detection and collision warnings in cars, to smartphones and electric vehicles that last longer between charges, to secure and reliable Wi-Fi and cellular connections.
January 2024
Microchip Technology Inc. got $162 million to increase its production of microcontroller units and other specialty semiconductors, and to support the modernization and expansion of fabrication facilities in Colorado Springs, CO, and Gresham, OR.
December 2023
The first of the CHIPS Act allocations, about $35 million, went to BAE Systems Inc., a federal government contractor. BAE was expected to use the money to help modernize an aging Nashua, NH facility and help quadruple the company’s production capacity for chips used in F-35 fighter jets.
Arm’s attempt to purchase part of Intel’s computer chip business may have been declined, but it once again demonstrates the important role played in the evolution of today’s tech by Apple and its handheld Newton.
While it isn’t quite clear what Arm would gain by buying Intel, Bloombergclaims it tried to do so all the same. (And Apple hasn’t).
What makes this a little confusing is that Arm and Intel have such different businesses: while Arm is a chip design house, Intel is both a designer and manufacturer that uses an architecture cross-licensed with AMD. It is also fair to observe (as The Register has here) that while Intel’s star may be fading, it’s still worth billions of dollars.
Is this a melting point?
All the same, the story reveals something else. It shows the extent to which the tech industry has been transformed by the mobile productivity philosophy articulated in the early days by Newton.
You see, Newton strove to be the perfect assistant for business users. It offered handwriting recognition, natural language support, and though there was no ChatGPT (or even Siri), it hinted at the pervasive AI entering daily existence today.
Newton also used a true mobile processor from Arm (then known as Acorn). This was capable of delivering computational performance for very low energy (at the time) — just like the A-series chips inside iPhones and some iPads, or the M-series silicon inside Macs and iPad Pro.
Getting the team together
The decision to work with Arm reflected Apple’s early recognition that energy consumption and computational performance would be vital if it were to overcome the barriers against next-generation computer design. When Steve Jobs returned to Apple, he closed the Newton project — but in other ways, he continued to lean into what the company had learned. The data detector technology used in Mail arguably owes a debt to the PDA, as does handwriting recognition.
Many of the ideas Newton heralded appeared a decade later in iPhone — eventually, including that Arm-based processor.
The Arm-based Apple Silicon chips now used across Apple’s products are so performant that competitors struggle to keep up. The closest current rival (at least in the low-energy, high-performance stakes) is Qualcomm, which may well be trying to snap at Apple’s heels and has been forced to adopt Arm reference designs to do so. Even Microsoft is moving to Arm, while Intel’s processor design unit is spending more than it makes, prompting pain across the company.
You could argue that all of this illustrates how ahead of its time the Apple Newton was, and the huge influence it still wields today. Because today we have fully networked, high-performing devices in a myriad of different forms (Mac, smartphone, tablet, watch, glasses, more) boosted by AI with user interfaces to match.
Arguably, many of these devices were made possible by design decisions Apple reached when building the Newton. It took decades to accomplish it, but that means we are now living in Newton’s future, and the chutzpah of Arm’s overture to Intel reflects this. While what Arm would gain from any such deal isn’t clear, the alleged attempt illustrates that truth. Several decades later, revenge seems sweet.
Microsoft has outlined plans to secure its controversial Windows Recall feature ahead of launch.
Microsoft unveiled Recall in May, pitching it as an “explorable timeline of your PC’s past.” When enabled, Recall will record all user actions on Copilot Plus PCs, taking “snapshots” of the screen at 5-second intervals. Users can then access a searchable timeline of everything they’ve interacted with on their device, whether that’s an application, website, document, image, or anything else.
Microsoft initially intended to release Recall in June as a flagship feature for its Copilot Plus PCs. Those plans were put on hold amid a data security and privacy backlash, as experts claimed it would create a treasure trove of data for hackers, with some comparing it keylogger malware.
On Friday, the company detailed some of the ways it will protect user data in a blog post spelling out its security architecture. As Microsoft previously stated, Recall is now opt-in, so Copilot Plus users must turn the feature on or it won’t be record their screen. Biometric authentication is also required via Windows Hello each time a user wants to turn Recall on.
Recall snapshots and related data are stored and encrypted on a user’s device, Microsoft said, and are protected by a “virtualization-based security enclave” (VBS Enclave).
“VBS Enclaves use the same hypervisor as Azure to segment the computer’s memory into a special protected area where information can be processed,” said David Weston, vice president for Enterprise and OS Security at Microsoft. “Using Zero Trust principles, code in these enclaves can use cryptographic attestation protocols to safeguard that the environment is secure before performing sensitive operations, such as snapshot processing.”
Microsoft will provide users with a range of privacy controls, said Weston. Users can delete recorded snapshots individually or in bulk over a specified period and select how long Recall content is retained, for example. Content filtering is on by default to help prevent Recall from taking screenshots of information such as credit card details and websites — health or finance-related sites, for instance.
“With the Recall controls, a user can store as much or as little as they would like and remain in control,” he said.
Microsoft didn’t specify a release date for Recall, and didn’t respond to a request for more information on the planned launch.
In a significant move, California Governor Gavin Newsom vetoed a highly contested artificial intelligence (AI) safety bill on Sunday, citing concerns over its potential to stifle innovation and impose excessive restrictions on AI technologies, including basic functions.
Newsom’s decision to veto Senate Bill 1047, which would have required safety testing and imposed stringent standards on advanced AI models, came after tech giants like Google, Meta, and OpenAI raised concerns that the bill could hamper AI innovation in the state and possibly push companies to relocate.
“While well-intentioned, SB 1047 (The AI Bill) does not take into account whether an AI system is deployed in high-risk environments, involves critical decision-making or the use of sensitive data,” Newsom said in a statement.
“Instead,” he added, “the bill applies stringent standards to even the most basic functions — so long as a large system deploys it. I do not believe this is the best approach to protecting the public from real threats posed by the technology.”
Newsom emphasized that while the bill was well-intentioned, it failed to differentiate between high-risk AI systems and those with less impact on public safety.
The bill, authored by Democratic State Senator Scott Wiener, sought to mandate safety testing for the most advanced AI models, particularly those costing more than $100 million to develop or requiring significant computing power. It also proposed creating a state entity to oversee “Frontier Models” — highly advanced AI systems that could pose a greater risk due to their capabilities.
Besides, the bill also required developers to implement a “kill switch” to deactivate models that pose a threat and to undergo third-party audits to verify their safety practices.
Proponents such as Senator Wiener argued that voluntary commitments from AI developers were insufficient and that enforceable regulations were necessary to protect the public from potential AI-related harm.
Senator Scott Wiener expressed disappointment, arguing that the lack of enforceable AI safety standards could put Californians at risk as AI systems continue to advance at a rapid pace.
“This veto is a setback for everyone who believes in oversight of massive corporations that are making critical decisions that affect the safety and welfare of the public and the future of the planet,” Senator Wiener said in a statement.
Wiener had earlier stated that voluntary commitments from AI companies were insufficient to ensure public safety, calling the veto a setback in efforts to hold powerful AI systems accountable.
“While the large AI labs have made admirable commitments to monitor and mitigate these risks, the truth is that voluntary commitments from industry are not enforceable and rarely work out well for the public,” Wiener added.
Earlier this month the California State Assembly had passed the bill only to go to Newsom to approve or veto.
Expanding the horizon
While Governor Newsom vetoed the AI bill, he also balanced the veto by announcing a series of new initiatives aimed at protecting Californians from the risks posed by fast-developing generative AI (GenAI) technology, the statement added.
The Governor has signed 17 bills related to generative AI technology, covering areas like AI-generated misinformation, deepfake prevention, AI watermarking, and protecting children and workers from harmful AI applications.
According to Newsom, this legislative package is the most comprehensive set of AI regulations in the country.
“We have a responsibility to protect Californians from potentially catastrophic risks of Gen AI deployment. We will thoughtfully — and swiftly — work toward a solution that is adaptable to this fast-moving technology and harnesses its potential to advance the public good,” Newsom said in the statement.
Among the new measures, Newsom has tasked California’s Office of Emergency Services (Cal OES) with expanding its assessment of risks posed by GenAI to the state’s critical infrastructure. This includes energy, water, and communications systems, to prevent mass casualty events. In the coming months, Cal OES will conduct risk assessments in collaboration with AI companies that develop frontier models and with infrastructure providers across various sectors.
Additionally, Newsom has directed state agencies to engage with leading AI experts to develop “empirical, science-based trajectory analysis” for AI systems, with a focus on high-risk environments.
He also announced plans to work closely with labor unions, academic institutions, and private sector stakeholders to explore the use of Gen AI technology in workplaces, ensuring that AI tools can benefit workers while maintaining safety and fairness.
Newsom’s veto and subsequent announcements underscore the state’s complex position as both a leader in AI innovation and a regulator of potentially disruptive technologies. While tech industry leaders, including Microsoft, OpenAI, and Meta, have opposed the bill, others, like Tesla CEO Elon Musk, have supported the bill, emphasizing the need for more stringent AI safeguards. OpenAI’s chief strategy officer Jason Kwon in a letter to Senator Weiner said the bill would “stifle innovation.”
The controversy surrounding AI regulation in California reflects broader national and global concerns about the impact of AI on society. As federal legislation on AI safety stalls in Congress, California’s actions are being closely watched by policymakers and industry leaders alike.
Despite vetoing SB 1047, Newsom signaled that further AI legislation could be on the horizon.
“A California-only approach may well be warranted — especially absent federal action by Congress,” Newsom said in the statement, leaving open the possibility of revisiting AI safety measures in future legislative sessions.
Checkr provides 1.5 million personnel background checks per month for thousands of businesses, a process that requires generative AI (genAI) and machine learning tools to sift through massive amounts of unstructured data.
The automation engine produces a report about each potential job prospect based on background information that can come from a number of sources, and it categorizes criminal or other issues described in the report.
Of Checkr’s unstructured data about 2% is considered “messy,” meaning the records can’t be easily processed with traditional machine learning automation software. So, like many organizations today, Checkr decided to try a genAI tool — in this case, OpenAI’s GPT-4 large language model (LLM).
GPT-4, however, only achieved an 88% accuracy rate on background checks, and on the messy data, that figure dropped to 82%. Those low percentages meant the records didn’t meet customer standards.
Checkr then added retrieval augmented generation (or RAG) to its LLM, which added more information to improve the accuracy. While that worked on the majority of records (with 96% accuracy rates), the numbers for more difficult data dropped even further, to just only 79%.
The other problem? Both the general purpose GPT-4 model and the one using RAG had slow response times: background checks took 15 and seven seconds, respectively.
So, Checkr’s machine learning team decided to go small and try out an open-source small language model (SLM). Vlad Bukhin, Checkr’s machine learning engineer, fine-tuned the SLM using data collected over years to teach what the company sought in employee background checks and verifications.
That move did the trick. The accuracy rate for the bulk of the data inched up to 97% — and for the messy data it jumped to 85%. Query response times also dropped to just half a second. Additionally, the cost to fine-tune an SLM based on Llama-3 with about 8 billion parameters was one-fifth of that for a 1.8 billion-parameter GPT-4 model.
To tweak its SLM, CheckR turned to Predibase, a company that offers a cloud platform through which Checkr takes thousands of examples from past background checks and then connects that data to Predibase. From there, the Predibase UI made it as easy as just clicking a few buttons to fine-tune the Llama-3 SLM. After a few hours of work, Bukhin had a custom model built.
Predibase operates a platform that enables companies to fine-tune SLMs and deploy them as a cloud service for themselves or others. It works with all types of SLMs, ranging in size from 300 million to 72 billion parameters.
SLMs have gained traction quickly and some industry experts even believe they’re already becoming mainstream enterprise technology. Designed to perform well for simpler tasks, SLMs are more accessible and easier to use for organizations with limited resources; they’re more natively secure, because they exist in a fully self-manageable environment; they can be fine-tuned for particular domains and data security; and they’re cheaper to run than LLMs.
Computerworld spoke with Bukhin and Predibase CEO Dev Rishi about the project, and the process for creating a custom SLM. The following are excerpts from that interview.
When you talk about categories of data used to perform background checks, and what you were trying to automate, what does that mean?Bukhin: “There are many different types of categorizations that we they would do, but in this case [we] were trying to understand what civil or criminal charges were being described in reports. For example, ‘disorderly conduct.'”
What was the challenge in getting your data prepared for use by an LLM? Bukhin: “Obviously, LLMs have only been popular for the past couple of years. We’ve been annotating unstructured data long before LLMs. So, we didn’t need to do a lot of data cleaning for this project, though there could be in the future because we are generating lots of unstructured data that we haven’t cleaned yet, and now that may be possible.”
Why did your initial attempt with GPT-4 fail? You started using RAG on an OpenAI model. Why didn’t it work as well as you’d hoped? Bukhin: “We tried GPT-4 with and without RAG for this use case, and it worked decently well for the 98% of the easy cases, but struggled with the 2% of more complex cases., was something I’d tried to fine tune before. RAG would go through our current training [data] set and it would pick up 10 examples of similarly categorized categories of queries we wanted, but these 2% [of complex cases, messy data] don’t appear in our training set. So that sample that we’re giving to the LLM wasn’t as effective.”
What did you feel failed? Bukhin: “RAG is useful for other use cases. In machine learning, you’re typically solving for the 80% or 90% of the problem, and then the longtail you handle more carefully. In this case where we are classifying text with a supervised model, it was kind of the opposite. I was trying to handle the last 2% — the unknown part. Because of that, RAG isn’t as useful because you’re bringing up known knowledge while dealing with the unknown 2%.”
Dev: “We see RAG be helpful for injecting fresh context into a given task. What Vlad is talking about is minority classes; things where you’re looking for the LLM to pick up on very subtle differences — in this case the classification data for background checks. In those cases, we find what’s more effective is teaching the model by example, which is what fine-tuning will do over a number of examples.”
Can you explain how you’re hosting the LLM and the background records? Is this SaaS or are you running this in your own data center? Bukhin: “This is where it’s more useful to use a smaller model. I mentioned we’re only classifying 2% of the data, but because we have a fairly large data lake that still is quite a few requests per second. Because our costs scale with usage, you have to think about the system set-up different. With RAG, you would need to give the model a lot of context and input tokens, which results in a very expensive and high latency model. Whereas with fine-tuning, because the classification part is already fine-tuned, you just give it the input. The number of tokens you’re giving it and that it’s churning out is so small that it becomes much more efficient at scale”
“Now I just have one instance that’s running and it’s not even using the full instance.”
What do you mean by “the 2% messy data” and what do you see as the difference between RAG and fine tuning? Dev: “The 2% refers to the most complex classification cases they’re working on.
“They have all this unstructured, complex and messy data they have to process and classify to automate the million-plus background checks they do every month for customers. Two percent of those records can’t process with their traditional machine learning models very well. That’s why he brought in a language model.
“That’s where he first used GPT-4 and the RAG process to try to classify those records to automate background checks, but they didn’t get good accuracy, which means those background checks don’t meet the needs of their customers with optimal occuracy.”
Vlad: “To give you an idea of scale, we process 1.5 million background checks per month. That results in one complex charge annotation request every three seconds. Sometimes that goes to several requests per second. That would be really tough to handle if it was a single instance LLM because it would just queue. It would probably take several seconds if you were using RAG on an LLM. It would take several seconds to answer that.
“In this case because it’s a small language model and it uses fewer GPUs, and the latency is less [under .15 seconds], you can accomplish more on a smaller instance.”
Do you have multiple SLMs running multiple applications, or just one running them all? Vlad: Thanks to the Predibase platform, you can launch several use cases solutions onto one [SLM] GPU instance. Currently, we just have the one, but there are several problems we’re trying to solve that we would eventually add. In Predibase terms, it’s called an Adapter. We would add another adatpersolution to the same model for a different use case.
“So, for example, if you’ve deployed a small language model like a Llama-3 and then we have an adapter solution on it that responds to one type of requests, we might have another adatper solution on that same instance because there’s still capacity, and itthat solution can respond to a completely different type of requests using the same base model.
“Same [SLM] instance but a different parameterized set that’s responsible just for your solution.”
Dev: “This implementation we’ve open-sourced as well. So, for any technologist that’s interested in how it works, we have an open-source serving project called LoRAX. When you fine-tune a model… the way I think about it is RAG just injects some additional context when you make a request of the LLM, which is really good for Q&A-style use cases, such that it can get the freshest data. But it’s not good for specializing a model. That’s where fine tuning comes in, where you specialized it by giving it sets of specific examples. There are a few different techniques people use in fine-tuning models.
“The most common technique is called LoRA, or low-rank adaptation. You customize a small percentage of the overall parameters of the model. So, for example, Llama-3 has 8 billion parameters. With LoRA, you’re usually fine tuning maybe 1% of those parameters to make the entire model specialized for the task you want it to do. You can really shift the model to be able to the task you want it to do.
“What organizations have traditionally had to do is put every fine-tuned model on its own GPU. If you had three different fine-tuned models – even if 99% of those models were the same – every single one would need to be on its own server. This gets very expensive very quickly.”
One of the things we did with Predibase is have a single Llama 3 instance with 8 billion parameters and bring multiple fine-tuned Adapters towards it. We call this small percentage of customized model weights Adapters because they’re the small part of the overall model that have been adapted for a specific task.
Vlad hasd a use case up now, let’s call it Blue, running on Llama 3 with 8 billion parameters that does the background classification. But if he had another use case, for example to be able to extract out key information in those checks, he could serve that same Adapter on top of his existing deployment.
This is essentially a way of building multiple use cases to be cost effective using the same GPU and base model.
How many GPU’s is Checkr using to run its SLM? “Vlad’s running on a single A100 GPU today.
“What we see is when using a small model version, like sub 8 billion-parameter models, you can run the entire model with multiple use cases on a single GPU, running on the Predibase cloud offering, which is a distributed cloud.”
What were the major differences between the LLM and the SLM? Bukhin: “I don’t know that I would have been able to run a production instance for this problem using GPT. These big models are very costly, and there’s always a tradeoff between cost and scale.
“At scale, when there are a lot of requests coming in, it’s just a little bit costly to run them over GPT. I think using a RAG situation, it was going to cost me about $7,000 per month using GPT, $12,000 if we didn’t use RAG but just asked GPT-4 directly.
“With the SLM, it costs about $800 a month.”
What were the bigger hurdles in implementing the genAI technology? Bukhin: “I’d say there weren’t a lot of hurdles. The challenge was as Predibase and other new vendors were coming up, there were still a lot of documentation holes and SDK holes that needed to be fixed so you could just run it.
“It’s so new that metrics were showing up as they needed to. The UI features weren’t as valuable. Basically, you had to do more testing on your own side after the model was built. You know, just debugging it. And, when it came to putting it into production, there were a few SDK errors we had to solve.
“Fine tuning the model itself [on Predibase] was tremendously easy. Parameter tuning was easy so we was just need to pick the right model.
“I found that not all models solve the problems with the same accuracy. We optimized with to Llama-3, but we’re constantly trying different models to see if we can get better performance, and better convergence to our training set.”
Even with small, fine-tuned models, users report problems, such as errors and hallucinations. What did you experience those issues, and how did you address them? Bukhin: Definitely. It hallucinates constantly. Luckily, when the problem is classification, you have the 230 possible responses. Quite frequently, amazingly, it comes up with responses that are not in that set of 230 possible [trained] responses. That’s so easy for me to check and just disregard and then redo it.
“It’s simple programmatic logic. This isn’t part of the small language model. In this context, we’re solving a very narrow problem: here’s some text. Now, classify it.
“This isn’t the only thing happening to solve the entire problem. There’s a fallback mechanism that happens… so, there are more models you try out and that that’s not working you try deep learning and then an LLM. There’s a lot of logic surrounding LLMs. There is logic that can help as guardrails. It’s never just the model. There’s programmatic logic around it.
“So, we didn’t need to do a lot of data cleaning for this project, though there could be in the future because we are generating lots of unstructured data that we haven’t cleaned yet, and now that may be possible. The effort to clean most of the data is already complete. But we could enhance some of the cleaning with LLMs”
