Tiny AI: The new oxymoron in town? Not really!

The last few days’ tech headlines were only about billions. And, of course, AI. Microsoft is all set to spend $17.5 billion in India (its largest investment in Asia). Some months back, Google also announced about investing $15 billion over the next five years to build an AI data centre (part of Google’s spending commitment of about $85 billion this year to build out data centre capacity). There is a Google $ 8 million funding package for four AI Centres of Excellence in India, too. Amazon is also in this BigTech pack- having announced plans to invest more than $35 billion across all its businesses in India through 2030. All this surely puts India in the global AI spotlight- as the confident and deserving hot zone on the global AI atlas.

But there are equally whopping numbers on the other side of these spreadsheets, too. The shadow beneath the spotlight could be our forte next if we leverage it timely and smartly. Something where we would need the right inch tape to begin with.

Micropsia or Macropsia

A study by Alex de Vries-Gao recently pointed out that greenhouse gas emissions from AI use are now equivalent to more than eight per cent of global aviation emissions. The 2025 carbon footprint of AI systems could be as high as 80 million tonnes, while the water used could reach 765 billion litres. Turns out that AI systems could be responsible for 32.6 to 79.7 million tonnes of CO₂ emissions per year. His analysis of sustainability reports from Apple, Google and Meta also showed that indirect water use is significantly underestimated and is probably a factor of three to four higher than the official estimate. AI systems can be assessed to use between 312.5 and 764.6 billion litres of water. Estimates suggest that the carbon footprint of AI systems could swing between 32.6 and 79.7 million tons of CO2 emissions in 2025. A World Energy Outlook report from the International Energy Agency had also observed that while a typical AI-focused data centre consumes as much electricity as 100 000 households, the largest ones under construction today will consume 20 times as much. Data centres took about 1.5 per cent of the world’s electricity consumption in 2024, or 415 terawatt-hours (TWh). And this is only going to move northwards with Data centre electricity consumption about to double to around 945 TWh by 2030 and to 1 200 TWh by 2035.

Ask Alex de Vries from Digiconomist (a research company dedicated to exposing the unintended consequences of digital trends), and he reminds us that if this pace is kept up, it won’t be long before AI systems will have the same carbon impact as global aviation. “My research is showing that the carbon and water footprints of AI are rapidly becoming more significant. The carbon emissions of AI systems last year were equivalent to around three per cent of the global carbon emissions of aviation. This year that percentage would have already increased to more than eight per cent.”

AI systems already consume more water than the entire global consumption of bottled water, while droughts as a result of climate change are becoming increasingly common. Investments in AI haven’t been slowing down, so we can expect further (rapid) escalation in the near future.

- Alex de Vries, Founder, Digiconomist

As AI gets big, it also brings in equally big demons that threaten our environment – be it electricity or power burdens or special silicon appetite or water guzzled in the AI pipes.

This looks like a new-age Alice in the AI wonderland. A scary and confusing rabbit-hole all over again. And maybe this time, too, the answer lies in something that makes her shrink.

The ‘Drink Me’ Bottle

Where’s that magical ‘Eat me’ cake? Could SLMs and minituarised models be the drink that would make today’s AI small enough to walk through these future doors without AI bumping into carbon-footprint issues? Would model compression tools like pruning, quantisation, and knowledge distillation help to lift some weight off the shoulders of heavy AI backyards? Lightweight models, edge devices that save compute resources, smaller algorithms that do not put huge stress on AI infrastructures, and AI that is thin on computational complexity- Tiny AI- as an AI creation and adoption approach- sounds unusual and promising at the onset. But how plausible would this approach be?

Owen Rogers, Uptime Institute Intelligence’s Senior Research Director of Cloud Computing, answers that by first drawing a picture of why AI is huge to begin with. “Most generative AI applications today rely on general-purpose large language models. General purpose is the operative term here: these models are designed to answer almost any question. To enable this broad applicability, the model must be trained on vast amounts of data, using extremely powerful hardware to turn that data into knowledge. When the model is used for inference, substantial power is needed to distil all that knowledge down into a useful response.”

Tiny AI is trained for a very specific task, without being bogged down in data it will never use.

- Owen Rogers, Uptime Institute

Tiny AI flips that approach, Rogers explains. “It’s trained for a very specific task, without being bogged down in data it will never use. This focus allows tiny models to be trained and run on much smaller hardware, using far less electricity.”

Would it not be great, then, if hardware innovations and new approaches to modelling that promote tiny AI work can be wielded here? That would be nice for relaxing compute and environmental weights of big AI infrastructures, right?

Yes, hardware innovations and new approaches to modelling that enable Tiny AI can significantly ease the compute and environmental burdens of large-scale AI infrastructures, avers Biswajeet Mahapatra, principal analyst at Forrester. “Specialised hardware like AI accelerators, neuromorphic chips, and edge-optimised processors reduces energy consumption by performing inference locally rather than relying on massive cloud-based models. At the same time, techniques such as model pruning, quantisation, knowledge distillation, and efficient architectures like transformers-lite allow smaller models to deliver high accuracy with far fewer parameters.”

This shift not only lowers carbon footprints but also democratises AI by making it viable on resource-constrained devices.

- Biswajeet Mahapatra, Principal analyst at Forrester

A lot is happening on the hardware front here.

Arun Chandrasekaran, Distinguished VP Analyst at Gartner, reckons that part. “Specialised low-power chips (NPUs, edge TPUs) and techniques like pruning, quantisation, and knowledge distillation enable models to run with less energy. By shifting workloads from centralised clouds to edge devices, Tiny AI cuts data movement, latency, and network-related emissions. While this does not eliminate the heavy compute required to train large foundation models, it sharply lowers the ongoing operational footprint.”

By shifting workloads from centralised clouds to edge devices, Tiny AI cuts data movement, latency, and network-related emissions.

- Arun Chandrasekaran, Distinguished VP Analyst at Gartner

Mobile device companies such as Apple, Google and Samsung are already bringing Tiny AI to mobile devices for optimised experiences, although their capability could be narrower than cloud-based large AI foundation models, he cites some actual progress happening in this area.

As to how it exactly works, there is a lot that changes under the hood. Ask Shrikant Acharya, co-founder and CTO, Excelfore (an end-to-end platform for the digital lifecycle management of edge devices in heterogeneous environments), and he says that Tiny AI has used jargon to define a ML based data acquisition model like our Excelfore’s eDataX, which provides data acquisition, filtering without sacrificing accuracy and building a knowledge base on anomaly detection. “We don’t call it ‘Tiny AI’, but the mathematical process is similar. Cost-effective, low-power microcontrollers are the preferred choice for zonal controllers capable of hosting AI workloads.

Inherently constrained compute and memory resources make conventional AI impractical, positioning Tiny AI as the natural fit for these environments.

- Shrikant Acharya, Co-founder and CTO, Excelfore

AI Compute will become cheap at some point, we just don’t know when, opines Hari Bayireddi, President, COO & Co-founder, Phenom (an HR technology company offering Applied AI in the Talent Management and Experience sphere). “Right now, the demand for AI infra is massive, but it will slow down at some stage- and I can say that for sure from my initial career experience of working in the server infrastructure business. At this point, we are at an experimental phase, and Small Language Models are solving the software part of huge AI, while orchestration can also play a big role in addressing huge AI’s concerns. It is an evolving industry.”

AI Compute will become cheap at some point, we just don’t know when.

- Hari Bayireddi, President, COO & Co-founder, Phenom

Not everyone feels likewise. When it comes to the hope that Tiny AI will ease the environmental concerns of big AI infrastructures, Vries feels differently. “This seems unlikely. The problem is that currently bigger is better, which means that bigger AI models perform better.” He explains further that AI is an umbrella term that includes a wide range of applications, so there are probably cases where a model is ‘good enough’ at a certain point. At the same time, in a highly competitive market where ChatGPT, Copilot, Gemini, DeepSeek, etc. are operating, it seems highly unlikely that the tech companies will be prepared to sacrifice performance for energy reductions.

The White Rabbit runs, again.

Tiny AI not only promises an AI paradigm that is lighter, leaner and greener but would also bring in happy side-effects like lower latency (Due to the use of edge devices), better privacy and control, apart from the much-needed democratisation of AI.

Mahapatra also opines that this shift not only lowers carbon footprints but also democratises AI by making it viable on resource-constrained devices, enabling real-time processing without heavy reliance on data centres. “Together, these innovations represent a critical step toward sustainable AI and broader adoption across industries.”

Tiny AI models run directly on edge devices, enabling fast, local decision-making by operating on narrowly optimised datasets and sending only relevant, aggregated insights upstream, Acharya spells out. “This approach delivers low latency and efficient bandwidth usage.”

At the same time, constraints stare in its face- at least, for now. There are hardware limitations, tiny memory resources that are difficult to accommodate deep learning models and cliffs related to the compiler and inference engine. As Acharya points out- The same constraints that make Tiny AI efficient also limit its flexibility - models cannot easily adapt to broader datasets or more complex use cases due to the restricted compute and memory available on microcontrollers.

Also, let’s not forget that in the past years, the big players have shown they have constantly used efficiency gains to keep making models larger (requiring more compute and energy), as Vries, who has played a major role in global discussion regarding the sustainability of blockchain technology and AI, reasons. “They are all competing for market share in this extremely competitive environment, so having the best model is a necessity. It’s also these large-scale applications that have primarily been driving up the power demand of AI systems, so if we can save some energy on several smaller applications that probably won’t matter too much in easing the environmental concerns surrounding big AI infrastructures.”

The answer, then, lies somewhere between gigantic AI skyscrapers and smart AI miniatures.

This is a good time for Alice to find those mushrooms that will help to get ‘just the right size’. Not too huge. Not too small. AI. Like a good grown-up.

pratimah@cybermedia.co.in