Architectures¶
| Purpose | Meaning | Major Application | Computation Complexity | Limitation | Advantage | |
|---|---|---|---|---|---|---|
| Linear Combinations | FC Fully-Connected | Poor scalability for large input sizes Do not capture “intuitive” invariances | ||||
| Spatial pattens | CNN (Convolutional) | - Require that activations between layers occur only in “local” manner - Treat hidden layers themselves as spatial images - Share weights across all spatial locations | Images, Videos | High | Reduce parameter count Capture [some] “natural” invariances | |
| Temporal (Sequences) | RNN (Recurrent) | Forward-feed, backward-feed, and self-loop is allowed | Time Series | |||
| GRU | Time Series | |||||
| LSTM | Time Series | |||||
| Transformer | Text Generation | |||||
| IDK | ResNet (Residual Network) | Add operator | Time Series | |||
| DenseNet | Concat operator | |||||
| U-Net | Basis of diffusion models Segmentation Super-Resolution Diffusion Models | |||||
| PINN (Physics-Informed) | ||||||
| Lagrangian | ||||||
| Deep Operator | ||||||
| Fourier Neural Operator | ||||||
| Graph Neural Networks |
IDK¶
Note¶
skip connections of inputs
- For structured problems
- DenseNet architecture
- The raw inputs may possess more meaningful information than the linear/non-linear combination
- Include concat of non-linear transformations, rather than pre-computing these transformations
- For unstructured problems
- ResNet is better