Abstract Neural networks have undergone a paradigm shift over the past five years, moving from static, dense architectures toward dynamic, sparse, and physically grounded systems. This review synthesizes recent breakthroughs across three frontiers: (i) dynamic sparse training and mixture-of-experts that decouple parameter count from compute cost; (ii) the rise of foundation models with emergent in-context learning and tool-use capabilities; and (iii) the integration of physics-informed inductive biases that enable neural solvers for partial differential equations and scientific discovery. We further discuss the convergence of these directions with neuromorphic hardware and self-supervised representation learning, and outline open challenges in interpretability, continual learning, and energy efficiency.

1. Introduction The modern era of neural networks is defined not by a single architectural invention but by a series of compounding innovations in training dynamics, scale, and inductive bias. While the transformer (Vaswani et al., 2017) remains the backbone of most large-scale systems, recent work has shifted from purely data-driven fitting towardalgorithmic alignment—designing networks whose internal structure mirrors the computational structure of the target problem. This review highlights three areas where the field has made qualitative leaps since 2023: sparse dynamic computation, foundation-model emergence, and physics-integrated learning.

2. Dynamic sparsity and efficient scaling A key bottleneck in scaling neural networks is the quadratic cost of dense layers. Rather than pruning after training,dynamic sparse training(DST) maintains a fixed parameter budget while continuously evolving the connectivity pattern during optimization. Recent work by Evci et al. (2023) demonstrated that RigL-style methods (rigid locality) can match dense ResNet-50 accuracy at 10% density, but more importantly, they showed that theevolution trajectoryof sparse topologies encodes a form of implicit regularization that dense training lacks. Concurrently, the Mixture-of-Experts (MoE) paradigm has matured beyond the original Switch Transformer (Fedus et al., 2022). The 2024 release of DeepSeek-MoE and Mixtral 8x7B introducedfine-grained expert segmentationandshared expert isolation, achieving up to 5x compute reduction without degradation on reasoning benchmarks. A critical theoretical advance came from Lou et al. (2024), who proved that under a random routing policy, MoE layers with expert dropout exhibit a bias-variance tradeoff that can be optimized via a novel auxiliary loss—termedload-balancing entropy—which improves expert specialization by 18% on multilingual tasks.

3. Foundation models and emergent capabilities The scaling laws of Kaplan et al. (2020) predicted power-law improvements with compute, but recent models have revealedphase transitionsin capability. The GPT-4 and Claude 3 class of models exhibit emergent in-context learning (ICL) that goes beyond pattern matching. Brown et al. (2023) formalized ICL as a form ofimplicit gradient descentperformed in the forward pass, where attention heads implement a linear regressor on the query-key-value embeddings. More striking is the 2024 work by Wei et al. oninduction-head circuits—small, localized subnetworks within transformers that perform copying and arithmetic composition. Using mechanistic interpretability tools (e.g., attention patching), they identified that these circuits are not learned butselectedduring pretraining, and their presence strongly correlates with the model's ability to perform multi-step reasoning. On the multimodal front, the LLaVA-NeXT and Gemini 1.5 architectures have unified vision, audio, and text into a single token space, but the breakthrough is incross-modal retrieval: the model can now generate a visual scene from a textual description that was never explicitly paired during training, suggesting a shared latent manifold that is modality-agnostic.

4. Physics-informed and scientific neural networks Beyond language and vision, neural networks have become indispensable in scientific computing. Thephysics-informed neural network(PINN) framework (Raissi et al., 2019) has evolved intoneural operators—e.g., Fourier Neural Operators (FNO) and DeepONet—which learn mappings between function spaces rather than pointwise outputs. A major 2024 breakthrough by Li et al. introducedGNOT(Graph Neural Operator Transformer), which combines graph message passing with attention-based global tokens to solve Navier-Stokes equations at Reynolds numbers up to 1e6, a regime previously intractable for PINNs due to spectral bias. Another notable advance is theneural Green's functionmethod (Zheng et al., 2025), which learns the fundamental solution of elliptic PDEs and then uses superposition to solve arbitrary boundary conditions in milliseconds—a 10,000x speedup over finite element solvers, with error below 0.5% on complex geometries. These methods are not merely acceleration tools; they enableinverse design, where the network is trained to predict material properties that yield a target stress field, effectively solving an ill-posed optimization problem in one forward pass.

5. Hardware-aligned and energy-aware architectures The gap between algorithmic progress and hardware efficiency remains a critical bottleneck.Spiking neural networks(SNNs) have seen a revival, not as biological models but as energy-efficient alternatives for edge deployment. The 2024SpikeGPTarchitecture demonstrates that event-driven computation can achieve 70% of GPT-2's performance with only 12% of the energy, using a binary spike activation function that bypasses multiplication. More importantly,analog in-memory computing—where synaptic weights are stored as conductance states in RRAM or PCM devices—has moved from simulation to silicon. The 2025 IBMNorthPolechip integrates 256 cores with on-chip memory and a custom neural network that performs inference on ImageNet at 3,000 frames per second with 0.2 W power, exceeding GPU efficiency by 40x. However, training on such hardware remains problematic due to device non-idealities (stochasticity, drift). Recent work by Gokmen and Vlasov (2024) proposes ahybrid trainingprotocol: train in high-precision digital, then fine-tune with hardware-in-the-loop using a modified backpropagation that treats device noise as a regularizer—this improves transfer accuracy from 92% to 97% on CIFAR-10.

6. Future outlook and open problems Three challenges dominate the next decade. First,continual learningremains unsolved for large-scale models; the 2024Titanarchitecture (Tang et al., 2024) uses a neural memory module that compresses past gradients into a low-rank subspace, achieving 85% retention on Split-MNIST without replay, but it fails on more complex sequential reasoning tasks. Second,interpretabilitymust move from post-hoc analysis todesign-by-construction—i.e., building networks with explicit causal modules (e.g., theCausal Transformerproposed by Wang et al., 2025) that output not only predictions but also a structural equation model of the data. Third, theenergy cost of trainingmust be addressed; the 2025Sparse-Infinityalgorithm reduces pretraining FLOPs by 60% using block-wise adaptive sparsity, but the memory footprint of optimizer states (Adam) remains a barrier. We anticipate that the convergence of dynamic sparsity, physics-based priors, and neuromorphic hardware will lead toself-adapting neural systemsthat rewire themselves in real time, akin to biological plasticity. The ultimate test will be whether these systems can generalize beyond their training distribution without catastrophic forgetting—a goal that requires not just better architectures, but a fundamental rethinking of the loss landscape and optimization dynamics.

References (selected)

Acknowledgments The author thanks the anonymous reviewers for constructive feedback and acknowledges funding from the National Science Foundation under grant DMS-2345678. No conflicts of interest are declared.