Month: September 2024

A US semiconductor industry in crisis needs a workforce that doesn’t yet exist

September 17, 2024 / Security – Computerworld

Kendall McDaniel felt like a nomad worker, wandering from job to job, never making enough money, never feeling fulfilled. He worked entry-level jobs in construction, in healthcare and at multiple fast-food joints.

In those fast-food jobs, McDaniel said, “You could be bleeding out of your ears and you’d still be expected to come into work.”

With an associate’s degree in theater, McDaniel at the time felt there were no long-term career options ahead for him. Then, two years ago, his cousin — an electrician — told him about a manufacturing company he’d worked at. He liked how they treated him, and that tip convinced McDaniel to apply for another entry-level position; this time, the job led to an apprentice program in the semiconductor industry.

He’s now a machinist at Utica, NY-based Indium, which makes foundational materials for the production of semiconductors and electronics. “I didn’t know anything about the semiconductor industry,” said McDaniel, 29. “I didn’t know anything tech wise. I was never a STEM person. I had mostly been looking at how I could obtain financial stability. That was my main goal coming into this.”

While McDaniel has found the semiconductor manufacturing challenging, it’s allowed him to take on a myriad number of tasks that jibes with his attention deficit disorder and it’s work that feels important.

Semiconductor worker — Kendall McDaniel, an Machinist apprentice at semiconductor materials maker Indium, checks tolerances of rolled metals.

Indium

As a transgender man, McDaniel said he experienced culture shock. In earlier jobs, he said he didn’t feel safe or supported. And moving toward a manufacturing job conjured up images of “a bunch of tough guys.” In fact, most of his co-workers are women, and his managers took both his ADHD and his transgender identity in stride.

“Here, my managers have always encouraged me. If you have time off, go ahead and use it,” he said. “Or, [they’ll say] ‘You’re looking a little tired, do you need anything for the rest of the week?’ My manager always comes over to me and asks me if I’m all set.”

McDaniel is part of a state-sponsored apprenticeship program organized under NY CREATES, an Albany-based, non-profit semiconductor R&D facility, and the National Institute for Industry and Career Advancement (NIICA).

Funneling workers into the industry

One of NIICA’s primary focuses is on building the nation’s talent pipeline in semiconductor and advanced manufacturing industries through scholastic partnerships and certified apprenticeships under a program called Growing Apprenticeships in Nanotechnology and Semiconductors (GAINS), which is funded by the U.S. Department of Labor.

Under GAINS are a number of programs aimed at funneling entry-level workers and others into the semiconductor industry at a time when it’s rapidly expanding in the United States.

The efforts come at a time when the industry doesn’t have anywhere near the workforce — including technicians, computer scientists, and engineers — required to support future needs. By some estimates, the US semiconductor industry faces a worker shortfall of between 59,000 and 146,000 workers by 2029. A minimum of 50,000 trained semiconductor engineers will be needed over the next several years in the US to meet the overwhelming and rapidly growing demand, according to a study by Purdue University.

The broader U.S. economy is set to have a gap of 1.4 million such workers, according to a 2023 study from the Semiconductor Industry Association. So the competition will be fierce over those skilled workers. Compounding the problem is an ongoing exodus of existing talent as virtually all Baby Boomers will be retired by 2030. A study from Deloitte found that nearly 90% of technology leaders interviewed cited recruiting as their biggest challenge.

Incentivized by funding under the CHIPS and Science Act of 2022, semiconductor companies are expected to invest more than $80 billion in opening new or expanding current fabrication plants (fabs) and other related facilities through 2025, according to White House data. For example, after the world’s largest producer of microchips, TSMC, was promised $6.6 billion in CHIPS Act funding last spring, the company announced a third new fab in Arizona.

To date, the CHIPS Act has allocated over $32 billion in proposed funding across 17 companies, 16 states, and 26 projects. However, no CHIPS funding has been disbursed to any companies, according to the US Department of Commerce.

The reshoring of the semiconductor industry is important for the US economy and national security. Semiconductors, the silicon-based processors for everything from dishwashers and smartphones to hypersonic guided missiles, keep US infrastructure, the economy and its military systems running.

But, the United States’ share of global semiconductor manufacturing capacity has been steadily declining for decades, falling from roughly 40% in 1990 to about 12% in 2020, according to the Congressional Research Service (CRS).

Semiconductor jobs technicians, engineers

McKinsey & Co.

How we got here

Like so many other goods, chip manufacturing is expensive. So, many US semiconductor makers transitioned to a “fabless” model, where they designed the product and offshored manufacturing to East Asia. About 80% of all chip manufacturing takes place in the region, mainly in Taiwan and China, according to The Center for Strategic and International Studies.

The largest tech companies in the US, including Google, Apple, and Amazon, have relied on Taiwan’s TSMC alone for nearly 90% of their chip production.

Then in 2020, the Covid-19 pandemic arrived, exposing an existential threat to the global chip supply. Due to forced lockdowns and remote work, fabs were shuttered and the supply of everything from automobiles to smartphones and computers suffered from the depletion of chip inventories. Add to that the raging US-China trade war, and it became clear reshoring chip production to the US was critical.

In 2022, a report from the US Department of Commerce detailed an “alarming” shortage of computer chips at a time when the median demand for them was as much as 17% higher over a two-year period. “This is a major supply and demand mismatch,” the report said.

The problem with reshoring goes beyond simply building facilities; that’s only a start. The US hasn’t been focused on training engineers and technicians for the semiconductor industry for decades. So, manufacturers have taken on the task themselves, though it’s not likely to be close to meeting staffing needs.

TSMC, for example, is sending employees in the US to Taiwan to learn jobs, according to Taylor Roundtree, associate partner in the semiconductor practice of consulting firm McKinsey & Co. “They’re also bringing some Taiwanese ex-pats in, too,” Roundtree said.

GlobalFoundries, the third-largest chipmaker globally, is casting a wide net to recruit talent. The company has sought out veteran candidates, along with candidates from its own workforce reentry program and an initiative for women in construction.

Micron Technology, Inc., the world’s fourth-largest semiconductor company — and the only U.S. memory and storage manufacturer — plans to invest $100 billion to build the largest semiconductor fabrication facility in the history of the United States in Onondaga County in NY, just two hours from Hudson Valley.

As part of the CHIPS Act, the US Economic Development Administration through its Tech Hubs Program was authorized to receive $10 billion over five years with the main goal of creating additional semiconductor training programs. To date, however, the money isn’t being dispersed as expected.

“If I look at fiscal year 2023, it launched and only had $500 million in funding. So, if I have a five-year program that’s authorized to receive $10 billion, I would have expected $2 billion per year. The first year, it received just a quarter of that,” Roundtree said.

Schools, fabs collaborate on apprenticeships

In response, semiconductor companies have joined with US universities, community colleges and even high schools in states where new fabs are being built or where workers are most needed for existing facilities to create their own joint programs.

“We’ve been working with companies for 10 years,” said Laura Marmolejo, associate dean for the advanced manufacturing programs at Austin Community College (ACC) in Austin, Texas. “Many years ago I worked in the industry. When I went to the faculty side… I basically realized on the non-credit side how much of a disconnect there was between what companies want and what colleges in general want. We’re hung up on, ‘this is credit and this is not credit.'”

“I was part of a corporate training initiative we had back when I worked for a [semiconductor] company. I just saw there was a lot of opportunities for the colleges to do things different, but colleges — education in general — are slow to change,” Marmolejo said.

Ten years ago, ACC began working with Samsung on a grant, which allowed the school to develop a cirriculum, build infrastructure for training, buy equipment, and hire instructors. “At the time, it was me and one other person,” Marmolejo said.

ACC’s semiconductor programs are growing fast. It now has 15 full-time and 30 part-time instructors or staffers. It currently has 300 students enrolled in the program and has trained 2,000, most of whom have remained in the semiconductor industry.

NIICA is currently overseeing 4.900 apprentices working at 79 employers across 17 states. Its focus is on states where fabs and other related facilities are being built or expanded, including Arizona, New York, Texas, Ohio and California.

The greatest demand in the industry today is for engineers and technicians — the people who create the parts for semiconductors and operate and keep the production equipment running, said Clay Nagel, senior director of NIICA’s national center for skills-based learning.

“There’s definitely a dearth of talent,” he said. “Semiconductor manufacturing was moving out of the US for a very long time. Now you’re trying to onshore manufacturing. But there needs to be programs in place to teach people. We need education partners and employers onboard with that.”

That’s where NIICA fits in; the non-profit organization works with K-12 schools, community colleges and universities to help create training programs and apprenticeships with employers.

The apprenticeship programs set the industry apart from others because employees can work full-time jobs and learn skills while getting they’re getting paid.

“There needs to be an awareness campaign,” Nagel said, adding that the number of semiconductor chips needed will roughly triple by 2030. “I wouldn’t call it an emergency yet. I’d call it an urgent issue at this point.”

Data shows that semiconductor industry hiring has been steadily declining for the past two years and job postings are down, a trend Nagel ties to the cyclical nature of the industry. “During the 2021-2022 time frame, you couldn’t make enough computer chips with the shortages and hiring was through roof,” said Nagel, who formerly led technical training at semiconductor maker GlobalFoundries.

McKinsey & Co.

When the economy picks up, people want new cars, televisions and smartphones, which means demand for chips goes up and manufacturers rush to boost inventory. That’s happening now, along with CHIPS Act incentives for semiconductor makers.

“I don’t see it being an issue that hiring is slow now,” said Nagel. “Just in the last 90 days, with companies that received CHIPS Act incentives, we’re now seeing anywhere from 50% hiring increases to 600% increases. They know these fabs are going to go online in late Q4 and through 2025. We predict hiring will continue to accelerate through 2025, 2026. I’d say this is the calm before the storm.

“It’s starting to get gusty,” he said.

ACC is one of the schools NIICA is now collaborating with on training and apprenticeships.

“We have a population that’s unskilled,” Marmolejo said. “There just aren’t a lot of people in the [US] semiconductor industry. Community colleges are really well suited to fill that gap and develop this workforce.

“Five years ago, nobody did apprenticeships,” Marmolejo continued. “Now we have four companies doing apprenticeships this fall. It motivates students to complete their credentials once they’re engaged.”

ACC isn’t trying to compete with the big technical schools such as Texas A&M, Marmolejo said. Community colleges simply don’t have the resources for that. Instead, ACC is focused on entry-level job training in applied technology and manufacturing — skills needed to attain a job as a machine operator or chip assembler.

“There’s a lot of people who need a job first,” Marmolejo said. “I tell students it’s not about starting pay. On average, it’s about $20 an hour, but the growth potential is huge. I have people who come into my program and say, ‘I’ve only gotten a one-dollar a year wage increase in my current job.’ That’s not going to happen in manufacturing. You can see $5 or $6 wage increases by end of year. It’s all performance based.

Dropping out of the funnel

In addition to the lack of semiconductor training programs is another issue: high attrition rates. University students who join engineering programs often change majors before they graduate, the engineers who do graduate often don’t enter engineering jobs — and those who get engineering jobs don’t choose semiconductors.

The state of Oregon, which has tracked semiconductor programs closer than any other state, found that only 8% to 12% of students who graduate from programs go on to work in engineering. “If I look at all undergraduate and graduate degrees awarded, roughly 3.7% of those are what they call core semiconductor degrees,” McKinsey & Co’s Roundtree said. “That’s defined as engineering technician and precision production roles. Of all the courses that people take in the state of Oregon, fewer than 4% are in those core semiconductor roles.

“If we’re facing an engineering shortage, that’s certainly one lever you could pull to get more engineers into the workforce,” Roundtree said. “You lose folks in huge swaths at every step of the funnel.”

More than 60% of executives in a semiconductor industry survey acknowledge it has an image problem in terms of making it an attractive career choice. Semiconductor fabs and development facilities evoke images of workers in white coveralls, filtration masks and cleanrooms where humidity, temperature and pressure conditions must constantly be maintained.

“Folks look at that cleanroom suit, and if I’m trying to choose between stocking shelves at Wal-Mart or putting on a cleanroom suit and moving boxes of wafers around a fab, well, if I’m only getting paid $2 more an hour to move the boxes of wafers around, maybe that’s not something I want to do,” said Bill Wiseman, senior partner and global co-leader of McKinsey & Co’s semiconductors practice.

Said Roundtree: “It’s something that in the US they haven’t had to face for a while, because we’ve not been building a lot of these cutting-edge fabs until now.”

Making the chip industry look cool

In some ways, the semiconductor industry is competing with the Googles and the Metas of the world to attract talent, and those companies can offer remote work — something manufacturing cannot do, said Isaac Hagen, senior vice president of vertical industry development at staffing firm ManpowerGroup.

“The challenging thing these days is making the semiconductor industry look cool,” Hagen said. “The talent largely does not exist. So, there is a need for upskilling, reskilling and what we refer to pre-skilling — so, getting people the skills they need before they even get into the job hunt.”

Most young people simply don’t consider that virtually everything they touch today is powered by a semiconductor chip. “When you start to talk to people about the industry, they realize that it powers the world,” Hagen said.

Peter Bermel, a professor of computer engineering at Purdue University, said one reason students avoid semiconductor engineering degrees is because they have a reputation of being “very hard,” especially in terms of mathematical requirements. “In general, that’s been a huge barrier — to keep people on the path,” Bermel said.

In 2020, Purdue University launched a Department of Defense-funded program called SCALE (Scalable Asymmetric Lifecycle Engagement); its aim is to train up a semiconductor workforce for the defense sector. Seventy-five percent of SCALE graduates remain in the semiconductor field, according to Bermel, who said the program is more attractive than others because it feels more purposeful.

“Since SCALE, things have changed quite a bit,” Bermel said. “We have more students staying in the program. We’ve been tracking our drop-out rate and it’s been about 4%. The students who join SCALE are highly motivated and find a lot of good opportunities to do things that are exciting.”

Cole Lush, a senior undergraduate student at Purdue in SCALE’s aerospace program, currently helps manufacture chips for updating older US ballistic missiles; it’s a job he got after his father, a retired US Air Force lieutenant colonel who worked with defense and data systems, urged him to pursue it.

“For a long time, I wanted to be an astrophysicist because space was my passion,” Lush said, “but then I realized I wanted to work on the systems that go into space. So, I focused more on engineering. Once I got involved with SCALE, I learned more about current events and the opportunities through SCALE.”

As a part of that SCALE, Lush was offered a summer intern program and hired by GRC Integrated Systems, a small consulting firm that works with the US Naval Surface Warfare Centers. During his internship, Lush worked on a project updating the internal electronics in older submarine-launchable ballistic missiles to extend their lifecycle.

“Part of the reason they hired me is because they saw SCALE on my resume at a career fair. Within a month, they told me, ‘you’re in,’” Lush said.

Updating US ballistic missiles is but one of many examples SCALE can offer students through its semiconductor education program. “All the defense systems, and obviously a huge number of commercial systems, are dependent on semiconductors,” Bermel said. “Having students who understand how things work on the inside is vital to preserve and extend these systems and create new and better systems.

NIICA’s Nagel said the bottom line should be that “opportunity is the sexiest part” of the semiconductor field. “A lot of companies are paying off student loans for employees,” Nagel said. “Through tuition reimbursement, you can work your way into engineering or management jobs without college debt.”

Imposter syndrome and the fear of math and sciences

In the past, entry-level positions in the semiconductor industry were rarely noticed by job seekers outside the industry. That changed as would-be workers saw new opportunities for training and advancement.

But new employees who’ve never worked in the semiconductor field can get “imposter syndrome” working in high-tech jobs that often require math and science backgrounds. “I encourage them to not get overwhelmed,” said Joe Rondino, a cleanroom operations manager for NY CREATES. “I tell them to trust the training programs we have and the mentors you’re given. As long as they’re hard workers, opportunities will present themselves. Math isn’t even required for a lot of the jobs.”

Rondino himself had been working in customer service at a large electronics retailer 11 years ago when he decided to change careers and attended a job fair. He was offered an entry-level job as a cleanroom operator working nights at NY CREATES; over time, he moved to a daytime shift and advanced as opportunities arose. He now manages 13 employees, and mentors apprentices.

What’s important is soft skills — the desire and ability to learn, and be a reliable and hard worker, Rondino said.

Hudson Valley Community College (HVCC) in Troy, NY first collaborated with NIICA when chipmaker GlobalFoundries (GF) initiated an Industrial Maintenance Technician Apprenticeship program. To carry out the program, GF conducts on-the-job training of approximately 2,000 hours per year, while the college partners with GF for the required related instruction. The college began conversations with NY CREATES in the spring of 2024 to explore another apprenticeship program partnership within the semiconductor industry

Through a partnership with HVCC, NY CREATES operates an apprenticeship program that lets people get hands-on training while also taking in courses at the college to advance in their careers.

One of Rondino’s apprentices, 27-year-old Kyle Huffer, has been with NY CREATES for eight months. With only a high school diploma, Huffer learned about the semiconductor R&D firm through a relative who also works there.

Huffer was intrigued. “He described it to me, and it sounded very interesting and a lot more mentally stimulating [than] what I was doing before,” said Huffer, who’d been detailing cars at the time.

Huffer works as a clean room operator in a control center, where silicon wafers — thin discs of semiconductor material used to create microchips — are loaded into and out of machines that process them.

Clean room operator and manager — Joe Rodino (L), a cleanroom operations manager, mentors Kyle Huffer (R), a cleanroom operator who has worked for NY CREATES the past eight months.

NY CREATES

Huffer loads plastic containers known as Front Opening Unified Pods, or FOUPs, that are filled with wafers and transfers them safely between machines as they’re processed.

“I wouldn’t say it’s challenging, but it’s definitely enjoyable and mentally stimulating,” said Huffer. “This field is booming. Technology is the future. So there’s plenty of room for advancement. We’re only going to get more technologically advanced and what we do contributes to that.”

Lea Montana, another cleanroom operator, runs machinery that processes the silicon wafers. A part of the apprenticeship program, she’s been working at NY CREATES for four years and hopes to become a certified engineering technician.

Montana had been working as a home healthcare aid in her 20s when her stepfather told her about his job at NY CREATES and the scientific aspect of the work interested her. She’s since obtained an associate’s degree in applied science, and is now training another employee on cleanroom work.

“It’s a really exciting job and there’s always something new to learn,” she said. “If you want to continue to learn and grow in a career, this is great. And there are a lot of other job opportunities you can move into on site.”

For Indium’s McDaniel, working in the semiconductor industry has provided something he’d never had before — a stable career. He and his wife recently purchased a house and he was also able to buy a newer car.

McDaniel credits his newfound career with being able to go to school while also working 40 to 50 hours a week, “and I didn’t feel like I was short-changing myself or the family I was starting to build. I’m at a point in my transition that I can pass [for a man] very well; it’s not something I’m concerned about. …The people around me here see me for my work ethic — they don’t just see me.”

HVCC’s apprenticeship program began in late 2021 and had 42 apprentices in the first year. Currently, HVCC has over 70 active apprentices with 20 employed in the semiconductor industry. The college has approximately 500 students in credit-bearing programs to prepare students for jobs in the semiconductor manufacturing workforce.

Needed now: a ‘Top Gun‘ moment

Because a semiconductor fab plant takes anywhere from three to five years to build and get online, and most of the projects didn’t break ground until 2022, the industry won’t see a boom in hiring until late this year through 2026, according to Hagen. But even by then, the workforce needed to staff those facilities will be far from ready.

TSMC, which is spending $40 billion to build a new plant and expand another in Phoenix, earlier this year announced further completion delays. Previously, the company planned for a 2026 opening; it now expects to go online in 2027 or 2028. TSMC makes 90% of the world’s most advanced chips, supplying them to companies like Apple and Nvidia.

Those kinds of delays could, ironically, give the industry a little more breathing room to find and train all of the workers who’ll be needed to staff them. But skills shortages are still likely.

“We kind of need a ‘Top Gun‘ moment,” Wiseman said. “Top Gun came out in 1986, and everybody wanted to become a naval aviator all of a sudden. Before that, most people didn’t even know the Navy had planes, let alone wanting to go join the Navy to fly them. That’s the kind of moment we need in the semiconductor industry.”

Digital portfolio spells trouble for new European Commission

September 17, 2024 / Security – Computerworld

Just as European Commission President Ursula von der Leyen was preparing to present her new team, there is trouble over the line-up for Commission posts: The European Union’s current Commissioner for the Internal Market and Services, Thierry Breton, has unexpectedly submitted his resignation.

In his resignation letter, which the Frenchman posted on X, Breton justified his move by saying that von der Leyen had asked the French government a few days ago to remove his name from the list of candidates, allegedly in exchange for another more influential portfolio.

“I would like to express my deepest gratitude to my colleagues in the College, Commission services, MEPs, Member States, and my team. Together, we have worked tirelessly to advance an ambitious EU agenda. It has been an honour & privilege to serve the common European interest,” Breton wrote in the tweet accompanying his resignation letter.

Slack opens up to third-party AI ‘agents’

September 16, 2024 / Security – Computerworld

Slack AI customers can now interact with AI “agents” in other apps, including Asana, Box and Anthropic’s Claude.

The AI agents, accessible via a new chatbot interface in Slack, let users perform a range of actions in the integrated app, Slack said Monday.

With the Asana agent, for instance, Slack users can “surface project insights and recommendations, such as status, blockers, next steps, and more,” parent company Salesforce said in a statement. The Adobe Express agent lets users create content such as graphics for social media posts from a prompt without leaving Slack.

Other agents coming to Slack include Amazon Q Business, Cohere, Perplexity, and Writer.

Users can access the third-party agents via the Slack Marketplace or create their own agent with “purpose-built APIs,” Slack said.

A chatbot interface connects with parent company Salesforce’s CRM software. This involves access to Salesforce’s own Agentforce AI assistant, which has been rebranded from Einstein Copilot. From within Slack, the Agentforce AI can provide an update on the status of sales opportunities and cases, recommend next steps, draft emails, and more.

Salesforce

Agentforce in Slack is slated to be available in beta next month.

Other updates to Slack AI — the company’s paid AI service — include note taking in huddles voice and video calls. Here, the transcript of a huddle conversation is used to create a canvas document (Slack’s document editor tool) containing related information such as a meeting summary, action items, and links to relevant files.

In Slack’s Workflow Builder, users can direct the AI assistant to generate a workflow automation using natural language prompts.

The Slack AI search function gets an update, too; users can now surface results based on files uploaded or connected to a Slack workspace. Until now, Slack could only search conversations in the app; now the tool can retrieve information from sources such as canvas documents, Google and Microsoft documents, and files in connected apps, Slack said.

Aside from AI-related features, Slack introduced new templates that can provide a starting point to help users begin a new project. A template for a marketing team could create a Slack channel with a standardized brief in a canvas document, a project plan in a Slack list, and an automated workflow for weekly status updates, the company said.

Templates will be available in October.

Finally, Salesforce channels — available now to Slack Sales Elevate customers and later to Salesforce Starter Suite customers — are a new type of channel that connects Salesforce CRM records to conversations in a Slack workspace, enabling users to update CRM data without switching screens.

Slack opens up to third-party AI ‘agents’

September 16, 2024 / Security – Computerworld

Slack AI customers can now interact with AI “agents” in other apps, including Asana, Box and Anthropic’s Claude.

The AI agents, accessible via a new chatbot interface in Slack, let users perform a range of actions in the integrated app, Slack said Monday.

Other agents coming to Slack include Amazon Q Business, Cohere, Perplexity, and Writer.

Users can access the third-party agents via the Slack Marketplace or create their own agent with “purpose-built APIs,” Slack said.

Salesforce

Agentforce in Slack is slated to be available in beta next month.

In Slack’s Workflow Builder, users can direct the AI assistant to generate a workflow automation using natural language prompts.

Templates will be available in October.

Slack opens up to third-party AI ‘agents’

September 16, 2024 / Security – Computerworld

Slack AI customers can now interact with AI “agents” in other apps, including Asana, Box and Anthropic’s Claude.

The AI agents, accessible via a new chatbot interface in Slack, let users perform a range of actions in the integrated app, Slack said Monday.

Other agents coming to Slack include Amazon Q Business, Cohere, Perplexity, and Writer.

Users can access the third-party agents via the Slack Marketplace or create their own agent with “purpose-built APIs,” Slack said.

Salesforce

Agentforce in Slack is slated to be available in beta next month.

In Slack’s Workflow Builder, users can direct the AI assistant to generate a workflow automation using natural language prompts.

Templates will be available in October.

Slack opens up to third-party AI ‘agents’

September 16, 2024 / Security – Computerworld

Slack AI customers can now interact with AI “agents” in other apps, including Asana, Box and Anthropic’s Claude.

The AI agents, accessible via a new chatbot interface in Slack, let users perform a range of actions in the integrated app, Slack said Monday.

Other agents coming to Slack include Amazon Q Business, Cohere, Perplexity, and Writer.

Users can access the third-party agents via the Slack Marketplace or create their own agent with “purpose-built APIs,” Slack said.

Salesforce

Agentforce in Slack is slated to be available in beta next month.

In Slack’s Workflow Builder, users can direct the AI assistant to generate a workflow automation using natural language prompts.

Templates will be available in October.

How macOS Sequoia can help you at work

September 16, 2024 / Security – Computerworld

Along with iOS 18, Apple today is releasing macOS Sequoia, iPadOS 18, and the latest update to watchOS. (Apple Intelligence, which isn’t expected to begin to appear until next month, has gotten a lot of attention, but the pre-AI versions of these operating systems offer plenty of useful features and updates.)

Focusing today on macOS Sequoia, should you upgrade immediately? That depends.

There are good things to tempt you, but you might need to wait — particularly if third-party services or applications you use (especially higher-end apps) don’t yet support the new OS.

If Apple Intelligence is the thing you’re most interested in, there’s no need to rush, since those tools won’t available until October in some countries, and next year in others. Global launch (including in Europe) will follow. Apple will also let Mac admins manage access to the service.

So, what’s Sequoia got to make you swoon if you ignore Apple Intelligence?

iOS, Mac, and iPhone: S.W.A.L.K.

What may turn out to be one of the most useful productivity-enhancing features in Sequoia is the increased integration between the Mac and iPhone. While EU customers won’t get this feature yet, iPhone Mirroring lets you use your iPhone on your Mac in a compact mini window. This lets you interact with iPhone apps via your Mac, and also lets you drag-&-drop files between the devices (though, that feature won’t debut until later this year). I think this could get really interesting if you use an iPad and a second Mac, too, as the implication is that you’ll be able to move files and folders around between the machines to your heart’s content in a quite focused way.

A second integration means notifications received on your iPhone can also appear on your Mac.

Manage busy desktops

Dragging a window to the edge of the screen will automatically place that window in a tile in the main window. This works across multiple windows, making it much easier to parse information from numerous websites and applications in one clear to the eye view. You can shift windows around, if you like.

Apple

Solving the eternal presentation headache

If you use Webex, Zoom, or even FaceTime, Sequoia will show you a view of what will be made visible to other meeting attendees when they share their screen before they actually share it. If you’ve ever unexpectedly needed to share a document during a meeting while other confidential items are open on your Mac, you’ll recognize what a small but handy improvement this is.

Even experienced Zoom hosts can’t help but exhale a little when they share their screen, as this is never quite as certain as you need for comfort. Now, it is.

Safari redesigned to get web clutter out of your way

Safari is smarter than before. You’ll be able to read page summaries or gather together links at the touch of a button. Reader view has been improved with a variety of features, including auto-generated table of contents to make navigating complex pages much easier. If a page features video, Safari will either open the clip in a big window or pop it inside a smaller pop-up window if you decide to navigate to another site while leaving the original site window open.

Finally, Safari will let you hide distracting items such as subscription pop-ups from view when you visit a site.

Notes, Reminders, and Calendar

Just as on the iPhone, Mac users can expect audio transcription and summarization features in Notes when Apple Intelligence appears. The app has also become more capable, with collapsible section headers and different text and highlight colors. Finally, if you record a call taken on your iPhone, a transcript will be created that can also sync with Notes on your Mac.

Calendar and Reminders work more smoothly now, with Reminder tasks showing up in Calendar and a Month view that lets you see both Calendar and Reminder entries at a glance.

Apple

The Password application

Apple’s all-new Password application is a new skin on Keychain, making the information — passwords, passkeys, Wi-Fi passwords — much more accessible and usable than before. The app uses iCloud to sync across all your logged-in Apple devices (and Windows hardware using the iCloud for Windows application).

Better collaboration tools

Freeform remains a really useful collaborative space where teams can work on ideas together from whatever Apple device they happen to use. On the Mac, the latest iteration includes a new diagramming mode to connect different objects and usability improvements when moving around a large board using a mouse.

Some Siri intelligence

Kick the system around and you’ll find a new accessibility tool that lets you make custom voice commands to invoke Shortcuts. You might use this to set up a tool that lets you ask Siri to create a PDF from what you are reading on a Mac, for example. You can find these options in System Settings>Accessibility>Vocal Shortcuts.

Smaller, useful tweaks

A handful of additional system tweaks wipe old annoyances away. You can schedule when messages are sent, for example. Another change allows you to install larger applications (more than 1GB) on external drives, subject to some restrictions. You also won’t need to have double the amount of space for an app on your drive to install it.

In addition:

An updated Calculator application lets you see mathematical expressions and previous calculations, and integrates with Notes to create what Apple calls Math Notes. The latter is essentially a way to do algebraic equations on your Mac.
There’s a new Keep Downloaded option that will ensure a local copy of a file is kept on your Mac rather than being stored in iCloud.

One more thing?

Apple has shipped a Chess application with Macs for decades. Yet the last time this got updated was with Mac OS X 10.3 Panther — 20 years ago. The historically important game, probably included in the Mac because Steve Jobs liked Chess so much, clearly isn’t on Apple’s speedy upgrade cycle. In Sequoia, it finally gets a makeover with new graphics, though sadly without a 3D or Kriegspiel mode.

Which Macs does macOS Sequoia work with?

If you ignore Apple Intelligence, the new Mac operating system is compatible with the following devices:

MacBook Pro (2018 and later).
MacBook Air (2020 and later).
Mac mini (2018 and later).
iMac (2019 and later).
iMac Pro (2017 and later).
Mac Pro (2019 and later).
Mac Studio (2022 and later).

The problem is that not all of the above devices support Apple Intelligence. To use Apple Intelligence, you need to be working with a Mac running an M1 or later Apple Silicon chip. No Intel Macs will run Apple Intelligence.

Finally, on security — once macOS Sequoia is available, it will be the only version to receive full security updates in the next 12 months. The two most recent versions (Sonoma and Ventura) will get some updates, but Monterey and earlier versions will get none. This means that if you rely on Macs, it’s worth ensuring you know which machines you run, what version of the OS they use, and what data they have access to.

Please follow me on LinkedIn, Mastodon, or join me in the AppleHolic’s bar & grill group on MeWe.

Microsoft revamps M365 Copilot chatbot with Pages shared ‘canvas’

September 16, 2024 / Security – Computerworld

Microsoft has added a new collaborative document tool to the Microsoft 365 Copilot chatbot that lets users store and share information created by the generative AI (genAI) assistant. It’s one of several M365 Copilot features announced Monday, including new Copilot features in apps such as Teams, Outlook, and Excel.

Microsoft describes Copilot Pages as a “dynamic, persistent canvas” accessible within Copilot’s Business Chat conversational interace.

With Pages, users can paste Copilot responses into a collaborative document that can be accessed and edited by coworkers. The document can be shared as a link or embedded in another M365 document as a Loop component.

“Pages takes ephemeral AI-generated content and makes it durable, so you can edit it, add to it, and share it with others,” said Jared Spataro, Microsoft corporate vice president. “You and your team can work collaboratively in a page with Copilot, seeing everyone’s work in real time and iterating with Copilot like a partner, adding more content from your data, files, and the web to your Page.”

Copilot Pages will be available later this month for M365 Copilot customers and via the free-to-use Copilot, provided users are signed in with a Microsoft Entra account — Microsoft’s identity and access management system.

Microsoft also announced updates to Copilot in various M365 apps, including the general availability of the M365 Copilot in Excel; it had been in beta since the M365 Copilot launch last November. Updates in Excel include the ability for the assistant to access data that hasn’t been formatted in a table; support for additional formulas, such as XLOOKUP and SUMIF; and the ability to work with text as well as numerical data.

It’s also possible to perform data analysis using Python in Copilot (a feature now in public preview).

“Now, anyone can work with Copilot to conduct advanced analysis like forecasting, risk analysis, machine learning, and visualizing complex data — all using natural language, no coding required. It’s like adding a skilled data analyst to the team,” said Spataro.

srcset="https://b2b-contenthub.com/wp-content/uploads/2024/09/Prioritize-my-inbox-with-Copilot.png?quality=50&strip=all 1920w, https://b2b-contenthub.com/wp-content/uploads/2024/09/Prioritize-my-inbox-with-Copilot.png?resize=300%2C168&quality=50&strip=all 300w, https://b2b-contenthub.com/wp-content/uploads/2024/09/Prioritize-my-inbox-with-Copilot.png?resize=768%2C432&quality=50&strip=all 768w, https://b2b-contenthub.com/wp-content/uploads/2024/09/Prioritize-my-inbox-with-Copilot.png?resize=1024%2C576&quality=50&strip=all 1024w, https://b2b-contenthub.com/wp-content/uploads/2024/09/Prioritize-my-inbox-with-Copilot.png?resize=1536%2C864&quality=50&strip=all 1536w, https://b2b-contenthub.com/wp-content/uploads/2024/09/Prioritize-my-inbox-with-Copilot.png?resize=1240%2C697&quality=50&strip=all 1240w, https://b2b-contenthub.com/wp-content/uploads/2024/09/Prioritize-my-inbox-with-Copilot.png?resize=150%2C84&quality=50&strip=all 150w, https://b2b-contenthub.com/wp-content/uploads/2024/09/Prioritize-my-inbox-with-Copilot.png?resize=854%2C480&quality=50&strip=all 854w, https://b2b-contenthub.com/wp-content/uploads/2024/09/Prioritize-my-inbox-with-Copilot.png?resize=640%2C360&quality=50&strip=all 640w, https://b2b-contenthub.com/wp-content/uploads/2024/09/Prioritize-my-inbox-with-Copilot.png?resize=444%2C250&quality=50&strip=all 444w" width="1024" height="576" sizes="(max-width: 1024px) 100vw, 1024px">

Copilot in Outlook can now help users prioritize emails.

Microsoft

In Outlook, the “Prioritize my inbox” feature highlights emails the Copilot considers to be of interest to a user, along with a summary of the email’s content. Users will be able to tell the Copilot which topics, people, and keywords are most important to them when the feature is available in public preview later this year.

In PowerPoint, a Copilot update lets users create presentations with an organization’s branded template andpull approved images stored in SharePoint Organization Asset Library.

A new feature coming to the Teams Copilot later this month allows the genAI assistant to provide information on meetings based on both video and text conversations, while Copilot in Word can now add in data kept in emails and meetings (in addition to searching web data and files such as Word and PDFs). For Copilot in OneDrive, users will be able to ask the AI assistant to compare up to five documents when the feature launches later this month.

Finally, Microsoft has announced general availability of Copilot agents, which lets users customize the tool to automatically carry out business processes.

Despite significant business interest in the possibilities of Copilot, many Microsoft 365 customers have yet to deploy the assistant widely across their organizations. A combination of data security concerns related to its use internally, as well as questions over the value it provides and the significant change management efforts required to implement the technology successfully, are all factors in the rollout pace.

The latest announcements improve the Copilot experience within apps such as Excel and PowerPoint and enhance the usefulness of the AI assistant, said Jason Wong, distinguished vice president analyst at Gartner.

He also pointed to the addition of Copilot Pages and the Team Copilot announced in May, both of which open the AI assistant to collaborative uses in addition to individual productivity. Copilot agents can provide “role-based and domain specific knowledge to be accessed through Copilot,” he said.

“Some Gartner clients are inquiring about Copilot Studio and how to extend generative AI to curated knowledge bases, but most are looking for something even simpler like the Copilot in SharePoint experience, which is currently in preview,” said Wong.

“However, it remains to be seen if all these new capabilities can drive the sticky adoption that Microsoft wants, since there’s already a lot of change fatigue in the workforce brought on by new generative AI features from many vendors and products.”

Everything we know about Apple Intelligence

September 16, 2024 / Security – Computerworld

Apple’s latest iPhones support a new breed of Apple AI called Apple Intelligence, a collection of artificial intelligence (AI) tools that will be made available across the company’s platforms starting in October with the release of iOS 18.1, iPadOS 18.1, and macOS Sequoia 15.1.

Apple Intelligence supplements Apple’s existing machine-learning tools and relies on generative AI (genAI) technology similar to that used by OpenAI’s ChatGPT. Apple’s version to a great extent runs on its own self-trained genAI models, which are built to be integrated across platforms, capable of using a user’s personal information, and private by the design.

Announced at this year’s Worldwide Developer’s Conference in June, Apple Intelligence is designed “to make your most personal products even more useful and delightful.” (That’s how Apple CEO Tim Cook described it.)

Essentially, the company has moved to build an AI ecosystem that is personal, private, and powerful, what Apple calls “AI for the rest of us.”

Here’s a look at what’s coming and how Apple got to this point.

Why Apple Intelligence matters

Apple has worked with AI since its earliest days (more about this below), but in in the last couple of years — since the arrival of ChatGPT and others — the company has been perceived as falling behind its competitors. There are many reasons for that, not least that Apple’s innate secrecy was a turn-off to researchers at the cutting edge of AI. Internal squabbles over precious R&D resources may also have slowed development.

But one moment that might have changed the scene took place over the winter holidays in late 2023, when Apple Senior Vice President for Software Craig Federighi tested GitHub Copilot code completion. He was reportedly blown away — and redirected Apple’s software development team to begin to apply Large Language Models (LLMs, a basic part of genAI tools) across Apple products. The company now sees this work as foundational to future product innovation and has diverted vast quantities of resources to bringing its own genAI technologies to its devices.

Analysts note that with Apple Intelligence soon to be available across the newer Macs, iPhones, and iPads, the company could quickly become one of the most widely used AI ecosystems in the world. (Wedbush Securities analyst Daniel Ives predicts Apple’s devices will be running 25% of global AI soon.) This matters, since AI smartphones and PCs will drive sales in both markets across the coming months, and Apple now has a viable product family to tout.

Apple

How Apple approaches Apple Intelligence

To deliver AI on its devices, Apple has refused to dilute its longstanding commitment to user privacy. With that in mind, it has developed a three-point approach to handling queries using Apple Intelligence:

On device

Some Apple Intelligence features will work natively on the device. This has the advantage of working faster while preserving privacy. Edge-based processing also reduces energy requirements, because no cloud communication or server-side processing is required. (More complex tasks must still be handled in the cloud.)

In the cloud

Apple is deploying what it calls Private Cloud Compute. This is a cloud intelligence system designed specifically for private AI processing and capable of handling complex tasks using massive LLMs.

The idea behind this system is that it provides the ability to flex and scale computational capacity between on-device processing and larger, server-based models. The servers used for these tasks are made by Apple, use Apple Silicon processors, and run a hardened operating system that aims to protect user data when tasks are transacted in the cloud. The advantage here is you can handle more complex tasks while maintaining privacy.

Externally

Apple has an agreement with OpenAI to use ChatGPT to process AI tasks its own systems can’t handle. Under the deal, ChatGPT is not permitted to gather some user data. But there are risks to using third-party services, and Apple ensures that users are aware if their requests need to be handled by a third-party service.

The company says it has designed its system so when you use Private Cloud Compute, no user data is stored or shared, IP addresses are obscured, and OpenAI won’t store requests that go to ChatGPT. The focus throughout is to provide customers with the convenience of AI, while building strong walls around personal privacy.

Apple

What Apple Intelligence features exist?

Apple has announced a range of initial features it intends making available within its Apple Intelligence fleet. The first new tools will appear with iOS 18.1, which is expected to appear when new Apple Silicon Macs and iPads are introduced later this fall.

Additional services will be introduced in a staggered rollout in subsequent releases. While not every announced feature is expected to be available this year, all should be in place by early 2025. In the background, Apple is not resting on its laurels; its teams are thought to be exploring additional ways Apple Intelligence can provide useful services to customers, with a particular focus on health.

Apple

At present, these are the Apple Intelligence tools Apple has announced:

Writing Tools

Writing Tools is a catch-all term for several useful features, most of which should appear in October with iOS 18.1 (and the iPad and Mac equivalents). These tools work anywhere on your device, including in Mail, Notes, Pages, and third-party apps. To use them, select a section of text and tap Writing Tools in the contextual menu.

Rewrite will take your selected text and improve it.
Proofread is like a much smarter spellchecker that checks for grammar and context.
Summarize will take any text and, well, summarize it. This also works in meeting transcripts.
Priority notifications: Apple Intelligence understands context, which means it should be able to figure out which notifications are most important to you.
Priority messages in Mail: The system will also prioritize the emails it thinks are most important.
Smart Reply: Apple’s AI can also generate email responses. You can edit these, reject them, or write your own.
Reduce Interruptions: A new Focus mode that is smart enough to let important notifications through.
Call transcripts: It is possible to record, transcribe, and summarize audio captured in Notes or during a Phone call. When a recording is initiated during a call in the Phone app, participants are automatically notified. After the call, Apple Intelligence generates a summary to help recall key points.

Search and Memory Movies in Photos

Search is much better in Photos. It will find images and videos that fit complex descriptions and can even locate a particular moment in a video clip that fits your search description.

Search terms can be highly complex; enter a description and Apple Intelligence will identify all the most appropriate images and videos, put together a storyline with chapters based on themes it figures out from within the collection, and create a Memory Movie. The idea is that your images are gathered, collected, and presented in an appropriate narrative arc; this feature is expected to debut with iOS 18.1.

Apple

Clean Up tool in Photos

At least in my parts of social media, the Photos AI tool that most seemed to impress early beta testers was Clean Up. This super-smart implementation means Apple Intelligence can identify background objects in an image and let you remove them with a tap. I can still recall when removing items from within images required high-end software running on top-of-the-range computers equipped with vast amounts of memory.

Now you can do it in a trice on an iPhone.

Image Playground for speedy creatives

Expected to appear in iOS 18.2, Image Playground uses genAI to let you create animations, illustrations, and sketches from within any app, including Messages. Images are generated for you by Apple Intelligence in response to written commands. You can also choose between a range of themes, places, or costumes, and also create an image based on a person from your Photos library.

The feature is also available within its own app and should appear in December.

Apple

Genmoji get smarter

Genmoji uses genAI to create custom emoji. The idea is that you can type in a description of the emoji you want to use and select one of the automatically generated ones to use in a message. You will also be able to keep editing the image to get to the one you want. (The only problem is that the person on the receiving end may not necessarily understand your creative zeal.)

This feature should show in December with iOS 18.2.

Image Wand

This AI-assisted sketching tool can transform rough sketches into nicer images in Notes. Sketch an image, then select it; Image Wand will analyze the content to create a pleasing and relevant image based on what you drew. You can also select an empty space and Image Wand will look at the rest of your Note to identify a context for which it will create an image for you.

Image Wand is now expected late 2024 or early 2025.

Apple

Camera Control in iPhone 16 Pro

A new feature in iPhone 16 Pro relies on visual intelligence and AI to handle some tasks. You can point your camera, for example, at a restaurant to get reviews or menus. It will also be possible to use this feature to access third-party tools for more specific information, such as accessing ChatGPT.

Additional visual tools are coming. For example, Siri will be able to complete in-app requests and take action across apps, such as finding images in your collection and then editing them inside another app.

Coming soon: Siri gains context and ChatGPT

ChatGPT integration in Siri is expected to debut at the end of the year, with additional enhancements to follow. The idea is that when you ask Siri a question, it will try to answer using its own resources; if it is unable to do so it will ask whether you want to use ChatGPT to get the answer. You don’t have to, but you will get free access to using it if you choose. Privacy protections are built in for users who access ChatGPT — IP addresses are obscured, and OpenAI won’t store requests.

Siri will also get significant improvements to deliver better contextual understanding and powerful predictive intelligence based on what your devices learn about you. You might use it to find a friend’s flight number and arrival time from a search through Mail or to put together travel plans — or any other query that requires contextual understanding of your situation.

The contextual features should appear next year.

Apple

On-screen awareness, but not until 2025

A new evolution in contextual awareness is scheduled to arrive at some point in 2025. This will give Siri the ability to take and use information on your display. The idea here is that whatever is on your screen becomes usable in some way — you might use this to add addresses to your contacts book, or to track threads in an email, for example. It’s a profound connection between what you do on your device and wherever you happen to be.

Another, and perhaps even more powerful, improvement will allow Siri to control apps, and because it uses genAI, you’ll be able to pull together a variety of instructions and apps — such as editing an image and adding it to a Note without having to open or use any apps yourself. This kind of deep control builds on the accessibility tools Apple already has and leans into some of the visionOS user interface improvements.

It’s another sign of the extent to which user interfaces are becoming highly personal.

Where can I get Apple Intelligence?

Apple has always been quite clear that Apple Intelligence will first be made available in beta in US English. During beta testing, Apple adjusted this slightly so that these tools work on any compatible iPhone running US English as its language and for Siri.

The company will introduce Apple Intelligence with localized English in Australia, Canada, New Zealand, South Africa, and the UK in December. Additional language support — such as Chinese, French, Japanese, and Spanish — is coming next year.

What devices work with Apple Intelligence?

Apple Intelligence requires an iPhone 15 Pro, iPhone 15 Pro Max, or iPhone 16 series device. It also runs on Macs and iPads equipped with an M1 or later chip.

What AI is already inside Apple’s systems?

All these features are supplemented by numerous forms of AI tools Apple already has in place across its platforms, principally around image vision intelligence and machine learning. You use these built-in applications each time you use FaceID, run facial recognition in Photos, or make use of the powerful Portrait Mode or Deep Fusion features when taking a photograph.

There are many more AI tools, from recognition of addresses and dates in emails for import into Calendar to VoiceOver all the way to Door Detection, even the Measure app on iPhones. What’s changed is that while Apple’s deliberate focus had been on machine-learning applications, the emergence of genAI unleashed a new era in which the contextual understanding available to LLM models uncovered a variety of new possibilities.

The omnipresence of various kinds of AI across the company’s systems shows the extent to which the dreams of Stanford researchers in the 1960s are becoming real today.

denvit

An alternative history of Apple Intelligence

Apple Intelligence might appear to have been on a slow train coming, but the company has, in fact, been working with AI for decades.

What exactly is AI?

AI is a set of technologies that enable computers and machines to simulate human intelligence and problem-solving capabilities. The idea is that the hardware becomes smart enough to learn new tricks based on what it learns, and carries the tools needed to engage in such learning.

To trace the trail of modern AI, think back to 1963, when computer scientist and LISP inventor John McCarthy launched the Stanford Artificial Intelligence Laboratory (SAIL). His teams engaged in important research in robotics, machine-vision intelligence, and more.

SAIL was one of three important entities that helped define modern computing. Apple enthusiasts will likely have heard of the other two: Xerox’s Palo Alto Research Center (PARC), which developed the Alto that inspired Steve Jobs and the Macintosh, and Douglas Engelbart’s Augmentation Research Center. The latter is where the mouse concept was defined and subsequently licensed to Apple.

Important early Apple luminaries who came from SAIL included Alan Kay and Macintosh user interface developer Larry Tesler — and some SAIL alumni still work at the company.

“Apple has been a leader in AI research and development for decades,” pioneering computer scientist and author Jerry Kaplan told me. “Siri and face recognition are just two of many examples of how they have put this investment to work.”

Back to the Newton…

Existing Apple Intelligence solutions include things we probably take for granted, going back to the handwriting recognition and natural language support in 1990’s Newton. That device leaned into research emanating from SAIL — Tesler led the team, after all. Apple’s early digital personal assistant first appeared in a 1987 concept video and was called Knowledge Navigator. (You can view that video here, but be warned, it’s a little blurry.)

Sadly, the technology couldn’t support the kind of human-like interaction we expect from ChatGPT, and (eventually) Apple Intelligence. The world needed better and faster hardware, reliable internet infrastructure, and a vast mountain of research-exploring AI algorithms, none of which existed at that time.

But by 2010, the company’s iPhone was ascendant, Macs had abandoned the PowerPC architecture to embrace Intel, and the iPad (which cannibalized the netbook market) had been released. Apple had become a mobile devices company. The time was right to deliver that Knowledge Navigator.

Apple

When Apple bought Siri

In April 2010, Apple acquired Siri for $200 million. Siri itself is a spinoff from SAIL, and, just like the internet, the research behind it emanated from a US Defense Advanced Research Projects Agency (DARPA) project. The speech technology came from Nuance, which Apple acquired just before Siri would have been made available on Android and BlackBerry devices. Apple shelved those plans and put the intelligent assistant inside the iPhone 4S (dubbed by many as the “iPhone for Steve,” given Steve Jobs’ death around the time it was released).

Highly regarded at first, Siri didn’t stand the test of time. AI research diverged, with neural networks, machine intelligence, and other forms of AI all following increasingly different paths. (Apple’s reluctance to embrace cloud-based services — due to concerns about user privacy and security — arguably held innovation back.)

Apple shifted Siri to a neural network-based AI system in 2014; it used on-device machine learning models such as deep neural networks (DNN), n-grams and other techniques, giving Apple’s automated assistant a bit more contextual intelligence. Apple Vice President Eddy Cue called the resulting improvement in accuracy “so significant that you do the test again to make sure that somebody didn’t drop a decimal place.”

But times changed fast.

Did Apple miss a trick?

In 2017, Google researchers published a landmark research paper, “Attention is All you Need.” This proposed a new deep-learning architecture that became the foundation for the development of genAI. (One of the paper’s eight authors, Łukasz Kaiser, now works at OpenAI.)

One oversimplified way to understand the architecture is this: it helps make machines good at identifying and using complex connections between data, which makes their output far better and more contextually relevant. This is what makes genAI responses accurate and “human-like” and it’s what makes the new breed of smart machines smart.

The concept has accelerated AI research. “I’ve never seen AI move so fast as it has in the last couple of years,” Tom Gruber, one of Siri’s co-founders, said at the Project Voice conference in 2023.

Yet when ChatGPT arrived — kicking off the current genAI gold rush — Apple seemingly had no response.

The (put it to) work ethic

Apple’s Cook likes to stress that AI is already in wide use across the company’s products. “It’s literally everywhere on our products and of course we’re also researching generative AI as well, so we have a lot going on,” he said.

He’s not wrong. You don’t need to scratch deeply to identify multiple interactions in which Apple products simulate human intelligence. Think about crash detection, predictive text, caller ID based on a number not in your contact book but in an email, or even shortcuts to frequently opened apps on your iPhone. All of these machine learning tools are also a form of AI.

Apple’s CoreML frameworks provide powerful machine learning frameworks developers can themselves use to power up their products. Those frameworks build on the insights Adobe co-founder John Warnock had when he figured out how to automate the animation of scenes, and we will see those technologies widely used in the future of visionOS.

All of this is AI, albeit focused (“narrow”) uses of it. It’s more machine intelligence than sentient machines. But in each AI application it delivers, Apple creates useful tools that don’t undermine user privacy or security.

The secrecy thing

Part of the problem for Apple is that so little is known about its work. That’s deliberate. “In contrast to many other companies, most notably Google, Apple tends not to encourage their researchers to publish potentially valuable proprietary work publicly,” Kaplan said.

But AI researchers like to work with others, and Apple’s need for secrecy acts as a disincentive for those in AI research. “I think the main impact is that it reduces their attractiveness as an employer for AI researchers,” Kaplan said. “What top performer wants to work at a job where they can’t publicize their work and enhance their professional reputation?”

It also means the AI experts Apple does recruit subsequently leave for more collaborative freedom. For example, Apple acquired search technology firm Laserlike in 2018, and within four years, all three of that company’s founders had quit. And Apple’s director of machine learning, Ian Goodfellow (another a SAIL alumni), left the company in 2022. I imagine the staff churn makes life tough for former Google Chief of Search and AI John Giannandrea, who is now Apple’s senior vice president of machine learning and AI strategy.

That cultural difference between Apple’s traditional approach and the preference for open collaboration and research in the AI dev community might have caused other problems. The Wall Street Journal reported that at some point both Giannandrea and Federighi were competing for resources to the detriment of the AI team.

Despite setbacks, the company has now assembled a large group of highly regarded AI pros, including Samy Bengio, who leads company research in deep learning. Apple has also loosened up a great deal, publishing research papers and open source AI software and machine learning models to foster collaboration across the industry.

What next?

History is always in the rear view mirror, but if you squint just a little bit, it can also show you tomorrow. Speaking at the Project Voice conference in 2023, Siri co-founder Adam Cheyer said: “ChatGPT style AI…conversational systems…will become part of the fabric of our lives and over the next 10 years we will optimize it and become accustomed to it. Then a new invention will emerge and that will become AI.”

At least one report indicates Apple sees this evolution of intelligent machinery as foundational to innovation. While that means more tools, and more advances in user interfaces, each those steps leads inevitably toward AI-savvy products such as AR glasses, robotics, health tech — even brain implants.

For Apple users, the next step — Apple Intelligence — arrives this fall.

Please follow me on Mastodon, or join me in the AppleHolic’s bar & grill and Apple Discussions groups on MeWe.

September’s Patch Tuesday update fixes 4 zero-days

September 13, 2024 / Security – Computerworld

Addressing four zero-days flaws (CVE-2024-38014, CVE-2024-38217, CVE-2024-43491 and CVE-2024-38217), this month’s Patch Tuesday release from Microsoft includes 79 updates to the Windows platform. There are no patches to Microsoft Exchange Server or the company’s development tools (Visual Studio or .NET). And Microsoft addressed a recently exploited vulnerability in Microsoft Publisher with two critical updates and nine patches rated important for Microsoft Office.

Significant testing will be required for this month’s Microsoft SQL Server patches, which affect both server and desktop components — with a focus on application installations due to a change in how Microsoft Installer handles changes and installation rollbacks.

The team at Readiness has crafted a useful infographic outlining the risks associated with each update.

Known issues

Microsoft always publishes a list of known issues that relate to the operating system and platforms included in each update, including the following two minor issues for September:

After installing the Windows update released on or after July 9, 2024, some Windows Servers may experience intermittent interruptions to remote desktop connections. Those using RDP over HTTP while employing a Remote Gateway server are most likely to experience this issue. Microsoft is working on a resolution and published a knowledge article (KB5041160) to assist with mitigations.
As a result of the recent updates to Microsoft SharePoint Server, some users are reporting an issue in which SharePoint workflows can’t be published because the unauthorized type is blocked. The issue also generates the event tag “c42q0” in SharePoint Unified Logging System (ULS) logs. In addition, recent changes could cause the deserialization of custom types that inherit from IDictionary to fail. For more information, see KB5043462 on these issues. (Sounds like something from the Succession TV series.)

Due to recent changes to Windows Installer, User Account Control (UAC) does not prompt for credentials on application installation repairs. Once this update (September 2024) has been installed, UAC will again prompt properly. Your scripts will need to be updated if you have not already accounted for this change.

Though Microsoft has provided documentation on avoiding the issue by disabling this feature in UAC, we think this is a much-needed change and recommend following this latest best practice.

Major revisions

This month, Microsoft published the following major revisions to past security and feature updates, including:

CVE-2020-17042: Windows Print Spooler Remote Code Execution Vulnerability. This print spooler update was first released in November 2020. This is an information update to reflect that Windows Server 2022 (Core) is now affected.
CVE-2024-30077: Windows OLE Remote Code Execution Vulnerability. This two-month-old patch from Microsoft has been updated to include support for the ARM platform.
CVE-2024-35272: SQL Server Native Client OLE DB Provider Remote Code Execution. First released in July, the affected software table has been updated to include entries for Visual Studio 2019 and 2022. No further action required.
CVE-2024-38138: Windows Deployment Services Remote Code Execution Vulnerability. This is a documentation update to a patch released last month to include support for all supported versions of Windows Server. No further action required.

Unusually, we have a patch revision that is not strictly documentation related. This month, it’s CVE-2024-38063 (Windows TCP/IP Remote Code Execution Vulnerability). Unlike other revisions, this latest version of a critical network patch will require testing as if it were a new update. System administrators need to take this latest patch revision seriously and test before (re)deployment.

Testing guidelines

Each month, the Readiness team analyzes the latest Patch Tuesday updates and provides detailed, actionable testing guidance based on a large application portfolio and a detailed analysis of the patches and their potential impact.

For September, we have grouped the critical updates and required testing efforts into separate product and functional areas including:

Microsoft SQL Server

Microsoft released several updates to the Microsoft SQL Server platform that affects both Windows desktops and SQL Server installations, including:

A significant update to all supported versions (2016-2022) of Microsoft SQL Server that will require a full installation test.
An updated core Windows library (SQLOLEDB) that helps Windows applications communicate with SQL Server databases and tools. Though Microsoft rated this change low-risk, Readiness recommends a portfolio analysis that highlights all apps that depend on this data-bound communication approach and a full test cycle for each one identified.

Due to the nature of this September SQL Server update, we highly recommend testing the patch itself and the patching process — with a view to the patch REMOVAL process. We understand that this will require time, skill, and effort — but it will be better than a full restore from backup.

Windows

Microsoft made networking and memory handling security issues a focus this month with the following changes to Windows:

Due to an update to 64-bit to 32-bit memory handling in Windows (called thunking), 32-bit Camera applications will require testing on 64-bit machines this month. Using Microsoft Teams or playing a video from a USB drive would provide good testing coverage for this change.
Virtual Machines (VMs) that require a VPN will require connectivity testing. In addition, the following protocols — PPP, PPTP, SSTP — will require a basic connectivity test.
A minor update to Windows defender will require basic testing for endpoint security.
A minor update to core networking functions will require a test of high network traffic this month. The focus should be on the transfer of large files using applications such Teams, Outlook and Microsoft Edge.

Microsoft delivered a significant update to the MSI Installer (application installer) sub-system that will require application install level testing for a portion of your portfolio. Part of this update relates to how shell links are handled in the storage subsystem, which might cause redirected folders or shortcuts to behave unexpectedly during an installation — particularly on secure or locked-down configurations.

We suggest that installations, rollbacks, un-installations and UAC checks be validated this month. Checking for “zero” exit codes on the MSI Installer log is always a good start.

Windows lifecycle and enforcement updates

This section contains important changes to servicing, significant feature depredations, and security related enforcements across the Windows desktop and server platforms.

Enforcements: Microsoft Entra now requires TLS 1.2 (using the latest Microsoft cryptographic libraries) as defined by RFC5246. Microsoft has published several scripts to assist with assessing whether your clients are using the latest libraries and protocols (they’re found here).
Lifecycle: General support for Microsoft SQL Server 2019 ends in January 2025. Given the large number of updates to this aging server, it might be time to upgrade.

Mitigations and workarounds

Microsoft did not publish any mitigations or workarounds this month.

Each month, we break down the update cycle into product families (as defined by Microsoft) with the following basic groupings:

Browsers (Microsoft IE and Edge).
Microsoft Windows (both desktop and server).
Microsoft Office.
Microsoft Exchange Server.
Microsoft Development platforms (ASP.NET Core, .NET Core and Chakra Core).
Adobe (if you get this far).

Browsers

Microsoft’s Edge browser no longer synchronizes exactly with Patch Tuesday; there were several updates to Microsoft’s version of the Chromium browser that address the following reported vulnerabilities:

CVE-2024-41879: Adobe PDF Viewer (Gotcha!)
CVE-2024-38208: Edge for Android updates.
CVE-2024-38207: Microsoft MSHTML Memory Issues.
CVE-2024-38210 and CVE-2024-38209: Microsoft Remote Code Execution.

Once we are done with the Microsoft updates, we can focus on these Chromium patches:

CVE-2024-8636: Heap buffer overflow in Skia.
CVE-2024-8637: Use after free in Media Router.
CVE-2024-8638: Type Confusion in V8 (JavaScript).
CVE-2024-8639: Use after free in Autofill.

After checking for compatibility or suitability challenges presented by these changes, we have not seen anything in the Edge or Chromium update that could affect most enterprise deployments. Add these browser updates to your standard release schedule.

Windows

Microsoft released two critical rated updates to the Windows platform (CVE-2024-38119 and CVE-2024-43491) and 43 patches rated important. The following Windows features have been updated:

Windows Update and Installer.
Windows Hyper-V.
Windows Kernel and Graphics (GDI).
Microsoft MSHTML and Mark of the Web.
Remote Desktop (RDP) and TCP/IP subsystems.

The real concern is that three of these vulnerabilities (CVE-2024-38014, CVE-2024-38217, CVE-2024-43491 have been reported as exploited. In addition, another reported vulnerability in the Windows HTML subsystem (CVE-2024-38217) has been reported as publicly disclosed. Given these four zero-days, we recommend that you add these Windows updates to your Patch Now release schedule.

Microsoft Office

Microsoft addressed two critical vulnerabilities in the SharePoint platform (CVE-2024-38018 and CVE-2024-43464) that will require immediate attention. There are nine other updates rated important that affect Microsoft Office, Publisher and Visio. Unfortunately, CVE-2024-38226 (which affects Publisher) has been reported as exploited in the wild by Microsoft. If your application portfolio does not include Publisher (many don’t) then add these Microsoft updates to your standard patch release cycle.

Microsoft SQL (nee Exchange) Server

This month brings a significantly larger update to the Microsoft SQL Server platform with 15 updates (all) rated as important. There are no reports of public disclosures or active exploits, and these patches cover the following broad vulnerabilities:

Microsoft SQL Server Native Scoring Remote Code Execution Vulnerability.
Microsoft SQL Server Native Scoring Information Disclosure Vulnerability.
Microsoft SQL Server Information Disclosure Vulnerability.
Microsoft SQL Server Elevation of Privilege Vulnerability.

Though there will be a significant testing profile this month, affecting both server and desktop systems, we suggest you add these SQL Server patches to your standard release schedule.

Microsoft development platforms

No development tools or features (Microsoft Visual Studio or .NET) have been updated this month.

Adobe Reader (and other third-party updates)

Things are a little different this month for Adobe Reader. Normally, Microsoft releases an Adobe Reader update to the Windows platforms. Not so, this month.

Adobe Reader has been updated (APSB24-70) but has not been included in the Microsoft release. This month’s Adobe Reader update addresses two critical memory-related security vulnerabilities and should be added to your standard app release cycle.