Topic: RNN (Recurrent Neural Networks) – Part 1 of 4: Introduction and Core Concepts
---
1. What is an RNN?
• A Recurrent Neural Network (RNN) is a type of neural network designed to process sequential data, such as time series, text, or speech.
• Unlike feedforward networks, RNNs maintain a memory of previous inputs using hidden states, which makes them powerful for tasks with temporal dependencies.
---
2. How RNNs Work
• RNNs process one element of the sequence at a time while maintaining an internal hidden state.
• The hidden state is updated at each time step and used along with the current input to predict the next output.
$$
h_t = \tanh(W_h h_{t-1} + W_x x_t + b)
$$
Where:
• $x_t$ = input at time step t
• $h_t$ = hidden state at time t
• $W_h, W_x$ = weight matrices
• $b$ = bias
---
3. Applications of RNNs
• Text classification
• Language modeling
• Sentiment analysis
• Time-series prediction
• Speech recognition
• Machine translation
---
4. Basic RNN Architecture
• Input layer: Sequence of data (e.g., words or time points)
• Recurrent layer: Applies the same weights across all time steps
• Output layer: Generates prediction (either per time step or overall)
---
5. Simple RNN Example in PyTorch
---
6. Summary
• RNNs are effective for sequential data due to their internal memory.
• Unlike CNNs or FFNs, RNNs take time dependency into account.
• PyTorch offers built-in RNN modules for easy implementation.
---
Exercise
• Build an RNN to predict the next character in a short string of text (e.g., “hello”).
---
#RNN #DeepLearning #SequentialData #TimeSeries #NLP
https://xn--r1a.website/DataScienceM
---
1. What is an RNN?
• A Recurrent Neural Network (RNN) is a type of neural network designed to process sequential data, such as time series, text, or speech.
• Unlike feedforward networks, RNNs maintain a memory of previous inputs using hidden states, which makes them powerful for tasks with temporal dependencies.
---
2. How RNNs Work
• RNNs process one element of the sequence at a time while maintaining an internal hidden state.
• The hidden state is updated at each time step and used along with the current input to predict the next output.
$$
h_t = \tanh(W_h h_{t-1} + W_x x_t + b)
$$
Where:
• $x_t$ = input at time step t
• $h_t$ = hidden state at time t
• $W_h, W_x$ = weight matrices
• $b$ = bias
---
3. Applications of RNNs
• Text classification
• Language modeling
• Sentiment analysis
• Time-series prediction
• Speech recognition
• Machine translation
---
4. Basic RNN Architecture
• Input layer: Sequence of data (e.g., words or time points)
• Recurrent layer: Applies the same weights across all time steps
• Output layer: Generates prediction (either per time step or overall)
---
5. Simple RNN Example in PyTorch
import torch
import torch.nn as nn
class BasicRNN(nn.Module):
def __init__(self, input_size, hidden_size, output_size):
super(BasicRNN, self).__init__()
self.rnn = nn.RNN(input_size, hidden_size, batch_first=True)
self.fc = nn.Linear(hidden_size, output_size)
def forward(self, x):
out, _ = self.rnn(x) # out: [batch, seq_len, hidden]
out = self.fc(out[:, -1, :]) # Take the output from last time step
return out
---
6. Summary
• RNNs are effective for sequential data due to their internal memory.
• Unlike CNNs or FFNs, RNNs take time dependency into account.
• PyTorch offers built-in RNN modules for easy implementation.
---
Exercise
• Build an RNN to predict the next character in a short string of text (e.g., “hello”).
---
#RNN #DeepLearning #SequentialData #TimeSeries #NLP
https://xn--r1a.website/DataScienceM
❤7
Topic: RNN (Recurrent Neural Networks) – Part 2 of 4: Types of RNNs and Architectural Variants
---
1. Vanilla RNN – Limitations
• Standard (vanilla) RNNs suffer from vanishing gradients and short-term memory.
• As sequences get longer, it becomes difficult for the model to retain long-term dependencies.
---
2. Types of RNN Architectures
• One-to-One
Example: Image Classification
A single input and a single output.
• One-to-Many
Example: Image Captioning
A single input leads to a sequence of outputs.
• Many-to-One
Example: Sentiment Analysis
A sequence of inputs gives one output (e.g., sentiment score).
• Many-to-Many
Example: Machine Translation
A sequence of inputs maps to a sequence of outputs.
---
3. Bidirectional RNNs (BiRNNs)
• Process the input sequence in both forward and backward directions.
• Allow the model to understand context from both past and future.
---
4. Deep RNNs (Stacked RNNs)
• Multiple RNN layers stacked on top of each other.
• Capture more complex temporal patterns.
---
5. RNN with Different Output Strategies
• Last Hidden State Only:
Use the final output for classification/regression.
• All Hidden States:
Use all time-step outputs, useful in sequence-to-sequence models.
---
6. Example: Many-to-One RNN in PyTorch
---
7. Summary
• RNNs can be adapted for different tasks: one-to-many, many-to-one, etc.
• Bidirectional and stacked RNNs enhance performance by capturing richer patterns.
• It's important to choose the right architecture based on the sequence problem.
---
Exercise
• Modify the RNN model to use bidirectional layers and evaluate its performance on a text classification dataset.
---
#RNN #BidirectionalRNN #DeepLearning #TimeSeries #NLP
https://xn--r1a.website/DataScienceM
---
1. Vanilla RNN – Limitations
• Standard (vanilla) RNNs suffer from vanishing gradients and short-term memory.
• As sequences get longer, it becomes difficult for the model to retain long-term dependencies.
---
2. Types of RNN Architectures
• One-to-One
Example: Image Classification
A single input and a single output.
• One-to-Many
Example: Image Captioning
A single input leads to a sequence of outputs.
• Many-to-One
Example: Sentiment Analysis
A sequence of inputs gives one output (e.g., sentiment score).
• Many-to-Many
Example: Machine Translation
A sequence of inputs maps to a sequence of outputs.
---
3. Bidirectional RNNs (BiRNNs)
• Process the input sequence in both forward and backward directions.
• Allow the model to understand context from both past and future.
nn.RNN(input_size, hidden_size, bidirectional=True)
---
4. Deep RNNs (Stacked RNNs)
• Multiple RNN layers stacked on top of each other.
• Capture more complex temporal patterns.
nn.RNN(input_size, hidden_size, num_layers=2)
---
5. RNN with Different Output Strategies
• Last Hidden State Only:
Use the final output for classification/regression.
• All Hidden States:
Use all time-step outputs, useful in sequence-to-sequence models.
---
6. Example: Many-to-One RNN in PyTorch
import torch.nn as nn
class SentimentRNN(nn.Module):
def __init__(self, input_size, hidden_size, output_size):
super(SentimentRNN, self).__init__()
self.rnn = nn.RNN(input_size, hidden_size, num_layers=1, batch_first=True)
self.fc = nn.Linear(hidden_size, output_size)
def forward(self, x):
out, _ = self.rnn(x)
final_out = out[:, -1, :] # Get the last time-step output
return self.fc(final_out)
---
7. Summary
• RNNs can be adapted for different tasks: one-to-many, many-to-one, etc.
• Bidirectional and stacked RNNs enhance performance by capturing richer patterns.
• It's important to choose the right architecture based on the sequence problem.
---
Exercise
• Modify the RNN model to use bidirectional layers and evaluate its performance on a text classification dataset.
---
#RNN #BidirectionalRNN #DeepLearning #TimeSeries #NLP
https://xn--r1a.website/DataScienceM
🔥2❤1
Topic: RNN (Recurrent Neural Networks) – Part 3 of 4: LSTM and GRU – Solving the Vanishing Gradient Problem
---
1. Problem with Vanilla RNNs
• Vanilla RNNs struggle with long-term dependencies due to the vanishing gradient problem.
• They forget early parts of the sequence as it grows longer.
---
2. LSTM (Long Short-Term Memory)
• LSTM networks introduce gates to control what information is kept, updated, or forgotten over time.
• Components:
* Forget Gate: Decides what to forget
* Input Gate: Decides what to store
* Output Gate: Decides what to output
• Equations (simplified):
---
3. GRU (Gated Recurrent Unit)
• A simplified version of LSTM with fewer gates:
* Update Gate
* Reset Gate
• More computationally efficient than LSTM while achieving similar results.
---
4. LSTM/GRU in PyTorch
---
5. When to Use LSTM vs GRU
| Aspect | LSTM | GRU |
| ---------- | --------------- | --------------- |
| Accuracy | Often higher | Slightly lower |
| Speed | Slower | Faster |
| Complexity | More gates | Fewer gates |
| Memory | More memory use | Less memory use |
---
6. Real-Life Use Cases
• LSTM – Language translation, speech recognition, medical time-series
• GRU – Real-time prediction systems, where speed matters
---
Summary
• LSTM and GRU solve RNN's vanishing gradient issue.
• LSTM is more powerful; GRU is faster and lighter.
• Both are crucial for sequence modeling tasks with long dependencies.
---
Exercise
• Build two models (LSTM and GRU) on the same dataset (e.g., sentiment analysis) and compare accuracy and training time.
---
#RNN #LSTM #GRU #DeepLearning #SequenceModeling
https://xn--r1a.website/DataScienceM
---
1. Problem with Vanilla RNNs
• Vanilla RNNs struggle with long-term dependencies due to the vanishing gradient problem.
• They forget early parts of the sequence as it grows longer.
---
2. LSTM (Long Short-Term Memory)
• LSTM networks introduce gates to control what information is kept, updated, or forgotten over time.
• Components:
* Forget Gate: Decides what to forget
* Input Gate: Decides what to store
* Output Gate: Decides what to output
• Equations (simplified):
f_t = σ(W_f · [h_{t-1}, x_t] + b_f)
i_t = σ(W_i · [h_{t-1}, x_t] + b_i)
o_t = σ(W_o · [h_{t-1}, x_t] + b_o)
C̃_t = tanh(W_C · [h_{t-1}, x_t] + b_C)
C_t = f_t * C_{t-1} + i_t * C̃_t
h_t = o_t * tanh(C_t)---
3. GRU (Gated Recurrent Unit)
• A simplified version of LSTM with fewer gates:
* Update Gate
* Reset Gate
• More computationally efficient than LSTM while achieving similar results.
---
4. LSTM/GRU in PyTorch
import torch.nn as nn
class LSTMModel(nn.Module):
def __init__(self, input_size, hidden_size, output_size):
super(LSTMModel, self).__init__()
self.lstm = nn.LSTM(input_size, hidden_size, batch_first=True)
self.fc = nn.Linear(hidden_size, output_size)
def forward(self, x):
out, (h_n, _) = self.lstm(x)
return self.fc(h_n[-1])
---
5. When to Use LSTM vs GRU
| Aspect | LSTM | GRU |
| ---------- | --------------- | --------------- |
| Accuracy | Often higher | Slightly lower |
| Speed | Slower | Faster |
| Complexity | More gates | Fewer gates |
| Memory | More memory use | Less memory use |
---
6. Real-Life Use Cases
• LSTM – Language translation, speech recognition, medical time-series
• GRU – Real-time prediction systems, where speed matters
---
Summary
• LSTM and GRU solve RNN's vanishing gradient issue.
• LSTM is more powerful; GRU is faster and lighter.
• Both are crucial for sequence modeling tasks with long dependencies.
---
Exercise
• Build two models (LSTM and GRU) on the same dataset (e.g., sentiment analysis) and compare accuracy and training time.
---
#RNN #LSTM #GRU #DeepLearning #SequenceModeling
https://xn--r1a.website/DataScienceM
👍1👎1
Topic: RNN (Recurrent Neural Networks) – Part 4 of 4: Advanced Techniques, Training Tips, and Real-World Use Cases
---
1. Advanced RNN Variants
• Bidirectional LSTM/GRU: Processes the sequence in both forward and backward directions, improving context understanding.
• Stacked RNNs: Uses multiple layers of RNNs to capture complex patterns at different levels of abstraction.
---
2. Sequence-to-Sequence (Seq2Seq) Models
• Used in tasks like machine translation, chatbots, and text summarization.
• Consist of two RNNs:
* Encoder: Converts input sequence to a context vector
* Decoder: Generates output sequence from the context
---
3. Attention Mechanism
• Solves the bottleneck of relying only on the final hidden state in Seq2Seq.
• Allows the decoder to focus on relevant parts of the input sequence at each step.
---
4. Best Practices for Training RNNs
• Gradient Clipping: Prevents exploding gradients by limiting their values.
• Batching with Padding: Sequences in a batch must be padded to equal length.
• Packed Sequences: Efficient way to handle variable-length sequences in PyTorch.
---
5. Real-World Use Cases of RNNs
• Speech Recognition – Converting audio into text.
• Language Modeling – Predicting the next word in a sequence.
• Financial Forecasting – Predicting stock prices or sales trends.
• Healthcare – Predicting patient outcomes based on sequential medical records.
---
6. Combining RNNs with Other Models
• RNNs can be combined with CNNs for tasks like video classification (CNN for spatial, RNN for temporal features).
• Used with transformers in hybrid models for specialized NLP tasks.
---
Summary
• Advanced RNN techniques like attention, bidirectionality, and stacked layers make RNNs powerful for complex tasks.
• Proper training strategies like gradient clipping and sequence packing are essential for performance.
---
Exercise
• Build a Seq2Seq model with attention for English-to-French translation using an LSTM encoder-decoder in PyTorch.
---
#RNN #Seq2Seq #Attention #DeepLearning #NLP
https://xn--r1a.website/DataScience4M
---
1. Advanced RNN Variants
• Bidirectional LSTM/GRU: Processes the sequence in both forward and backward directions, improving context understanding.
• Stacked RNNs: Uses multiple layers of RNNs to capture complex patterns at different levels of abstraction.
nn.LSTM(input_size, hidden_size, num_layers=2, bidirectional=True)
---
2. Sequence-to-Sequence (Seq2Seq) Models
• Used in tasks like machine translation, chatbots, and text summarization.
• Consist of two RNNs:
* Encoder: Converts input sequence to a context vector
* Decoder: Generates output sequence from the context
---
3. Attention Mechanism
• Solves the bottleneck of relying only on the final hidden state in Seq2Seq.
• Allows the decoder to focus on relevant parts of the input sequence at each step.
---
4. Best Practices for Training RNNs
• Gradient Clipping: Prevents exploding gradients by limiting their values.
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
• Batching with Padding: Sequences in a batch must be padded to equal length.
• Packed Sequences: Efficient way to handle variable-length sequences in PyTorch.
packed_input = nn.utils.rnn.pack_padded_sequence(input, lengths, batch_first=True)
---
5. Real-World Use Cases of RNNs
• Speech Recognition – Converting audio into text.
• Language Modeling – Predicting the next word in a sequence.
• Financial Forecasting – Predicting stock prices or sales trends.
• Healthcare – Predicting patient outcomes based on sequential medical records.
---
6. Combining RNNs with Other Models
• RNNs can be combined with CNNs for tasks like video classification (CNN for spatial, RNN for temporal features).
• Used with transformers in hybrid models for specialized NLP tasks.
---
Summary
• Advanced RNN techniques like attention, bidirectionality, and stacked layers make RNNs powerful for complex tasks.
• Proper training strategies like gradient clipping and sequence packing are essential for performance.
---
Exercise
• Build a Seq2Seq model with attention for English-to-French translation using an LSTM encoder-decoder in PyTorch.
---
#RNN #Seq2Seq #Attention #DeepLearning #NLP
https://xn--r1a.website/DataScience4M
Topic: 25 Important RNN (Recurrent Neural Networks) Interview Questions with Answers
---
1. What is an RNN?
An RNN is a neural network designed to handle sequential data by maintaining a hidden state that captures information about previous elements in the sequence.
---
2. How does an RNN differ from a traditional feedforward neural network?
RNNs have loops allowing information to persist, while feedforward networks process inputs independently without memory.
---
3. What is the vanishing gradient problem in RNNs?
It occurs when gradients become too small during backpropagation, making it difficult to learn long-term dependencies.
---
4. How is the hidden state in an RNN updated?
The hidden state is updated at each time step using the current input and the previous hidden state.
---
5. What are common applications of RNNs?
Text generation, machine translation, speech recognition, sentiment analysis, and time-series forecasting.
---
6. What are the limitations of vanilla RNNs?
They struggle with long sequences due to vanishing gradients and cannot effectively capture long-term dependencies.
---
7. What is an LSTM?
A type of RNN designed to remember long-term dependencies using memory cells and gates.
---
8. What is a GRU?
A Gated Recurrent Unit is a simplified version of LSTM with fewer gates, making it faster and more efficient.
---
9. What are the components of an LSTM?
Forget gate, input gate, output gate, and cell state.
---
10. What is a bidirectional RNN?
An RNN that processes input in both forward and backward directions to capture context from both ends.
---
11. What is teacher forcing in RNN training?
It’s a training technique where the actual output is passed as the next input during training, improving convergence.
---
12. What is a sequence-to-sequence model?
A model consisting of an encoder and decoder RNN used for tasks like translation and summarization.
---
13. What is attention in RNNs?
A mechanism that helps the model focus on relevant parts of the input sequence when generating output.
---
14. What is gradient clipping and why is it used?
It's a technique to prevent exploding gradients by limiting the gradient values during backpropagation.
---
15. What’s the difference between using the final hidden state vs. all hidden states?
Final hidden state is used for classification, while all hidden states are used for sequence generation tasks.
---
16. How do you handle variable-length sequences in RNNs?
By padding sequences to equal length and optionally using packed sequences in frameworks like PyTorch.
---
17. What is the role of the hidden size in an RNN?
It determines the dimensionality of the hidden state vector and affects model capacity.
---
18. How do you prevent overfitting in RNNs?
Using dropout, early stopping, regularization, and data augmentation.
---
19. Can RNNs be used for real-time predictions?
Yes, especially GRUs due to their efficiency and lower latency.
---
20. What is the time complexity of an RNN?
It is generally O(T × H²), where T is sequence length and H is hidden size.
---
21. What are packed sequences in PyTorch?
A way to efficiently process variable-length sequences without wasting computation on padding.
---
22. How does backpropagation through time (BPTT) work?
It’s a variant of backpropagation used to train RNNs by unrolling the network through time steps.
---
23. Can RNNs process non-sequential data?
While possible, they are not optimal for non-sequential tasks; CNNs or FFNs are better suited.
---
24. What’s the impact of increasing sequence length in RNNs?
It makes training harder due to vanishing gradients and higher memory usage.
---
25. When would you choose LSTM over GRU?
When long-term dependency modeling is critical and training time is less of a concern.
---
#RNN #LSTM #GRU #DeepLearning #InterviewQuestions
https://xn--r1a.website/DataScienceM
---
1. What is an RNN?
An RNN is a neural network designed to handle sequential data by maintaining a hidden state that captures information about previous elements in the sequence.
---
2. How does an RNN differ from a traditional feedforward neural network?
RNNs have loops allowing information to persist, while feedforward networks process inputs independently without memory.
---
3. What is the vanishing gradient problem in RNNs?
It occurs when gradients become too small during backpropagation, making it difficult to learn long-term dependencies.
---
4. How is the hidden state in an RNN updated?
The hidden state is updated at each time step using the current input and the previous hidden state.
---
5. What are common applications of RNNs?
Text generation, machine translation, speech recognition, sentiment analysis, and time-series forecasting.
---
6. What are the limitations of vanilla RNNs?
They struggle with long sequences due to vanishing gradients and cannot effectively capture long-term dependencies.
---
7. What is an LSTM?
A type of RNN designed to remember long-term dependencies using memory cells and gates.
---
8. What is a GRU?
A Gated Recurrent Unit is a simplified version of LSTM with fewer gates, making it faster and more efficient.
---
9. What are the components of an LSTM?
Forget gate, input gate, output gate, and cell state.
---
10. What is a bidirectional RNN?
An RNN that processes input in both forward and backward directions to capture context from both ends.
---
11. What is teacher forcing in RNN training?
It’s a training technique where the actual output is passed as the next input during training, improving convergence.
---
12. What is a sequence-to-sequence model?
A model consisting of an encoder and decoder RNN used for tasks like translation and summarization.
---
13. What is attention in RNNs?
A mechanism that helps the model focus on relevant parts of the input sequence when generating output.
---
14. What is gradient clipping and why is it used?
It's a technique to prevent exploding gradients by limiting the gradient values during backpropagation.
---
15. What’s the difference between using the final hidden state vs. all hidden states?
Final hidden state is used for classification, while all hidden states are used for sequence generation tasks.
---
16. How do you handle variable-length sequences in RNNs?
By padding sequences to equal length and optionally using packed sequences in frameworks like PyTorch.
---
17. What is the role of the hidden size in an RNN?
It determines the dimensionality of the hidden state vector and affects model capacity.
---
18. How do you prevent overfitting in RNNs?
Using dropout, early stopping, regularization, and data augmentation.
---
19. Can RNNs be used for real-time predictions?
Yes, especially GRUs due to their efficiency and lower latency.
---
20. What is the time complexity of an RNN?
It is generally O(T × H²), where T is sequence length and H is hidden size.
---
21. What are packed sequences in PyTorch?
A way to efficiently process variable-length sequences without wasting computation on padding.
---
22. How does backpropagation through time (BPTT) work?
It’s a variant of backpropagation used to train RNNs by unrolling the network through time steps.
---
23. Can RNNs process non-sequential data?
While possible, they are not optimal for non-sequential tasks; CNNs or FFNs are better suited.
---
24. What’s the impact of increasing sequence length in RNNs?
It makes training harder due to vanishing gradients and higher memory usage.
---
25. When would you choose LSTM over GRU?
When long-term dependency modeling is critical and training time is less of a concern.
---
#RNN #LSTM #GRU #DeepLearning #InterviewQuestions
https://xn--r1a.website/DataScienceM
❤4
Machine Learning
Photo
# 📚 PyTorch Tutorial for Beginners - Part 4/6: Sequence Modeling with RNNs, LSTMs & Attention
#PyTorch #DeepLearning #NLP #RNN #LSTM #Transformer
Welcome to Part 4 of our PyTorch series! This comprehensive lesson dives deep into sequence modeling, covering recurrent networks, attention mechanisms, and transformer architectures with practical implementations.
---
## 🔹 Introduction to Sequence Modeling
### Key Challenges with Sequences
1. Variable Length: Sequences can be arbitrarily long (sentences, time series)
2. Temporal Dependencies: Current output depends on previous inputs
3. Context Preservation: Need to maintain long-range relationships
### Comparison of Approaches
| Model Type | Pros | Cons | Typical Use Cases |
|------------------|---------------------------------------|---------------------------------------|---------------------------------|
| RNN | Simple, handles sequences | Struggles with long-term dependencies | Short time series, char-level NLP |
| LSTM | Better long-term memory | Computationally heavier | Machine translation, speech recognition |
| GRU | LSTM-like with fewer parameters | Still limited context | Medium-length sequences |
| Transformer | Parallel processing, global context | Memory intensive for long sequences | Modern NLP, any sequence task |
---
## 🔹 Recurrent Neural Networks (RNNs)
### 1. Basic RNN Architecture
### 2. The Vanishing Gradient Problem
RNNs struggle with long sequences due to:
- Repeated multiplication of small gradients through time
- Exponential decay of gradient information
Solutions:
- Gradient clipping
- Architectural changes (LSTM, GRU)
- Skip connections
---
## 🔹 Long Short-Term Memory (LSTM) Networks
### 1. LSTM Core Concepts
Key Components:
- Forget Gate: Decides what information to discard
- Input Gate: Updates cell state with new information
- Output Gate: Determines next hidden state
### 2. PyTorch Implementation
#PyTorch #DeepLearning #NLP #RNN #LSTM #Transformer
Welcome to Part 4 of our PyTorch series! This comprehensive lesson dives deep into sequence modeling, covering recurrent networks, attention mechanisms, and transformer architectures with practical implementations.
---
## 🔹 Introduction to Sequence Modeling
### Key Challenges with Sequences
1. Variable Length: Sequences can be arbitrarily long (sentences, time series)
2. Temporal Dependencies: Current output depends on previous inputs
3. Context Preservation: Need to maintain long-range relationships
### Comparison of Approaches
| Model Type | Pros | Cons | Typical Use Cases |
|------------------|---------------------------------------|---------------------------------------|---------------------------------|
| RNN | Simple, handles sequences | Struggles with long-term dependencies | Short time series, char-level NLP |
| LSTM | Better long-term memory | Computationally heavier | Machine translation, speech recognition |
| GRU | LSTM-like with fewer parameters | Still limited context | Medium-length sequences |
| Transformer | Parallel processing, global context | Memory intensive for long sequences | Modern NLP, any sequence task |
---
## 🔹 Recurrent Neural Networks (RNNs)
### 1. Basic RNN Architecture
class VanillaRNN(nn.Module):
def __init__(self, input_size, hidden_size, output_size):
super().__init__()
self.hidden_size = hidden_size
self.rnn = nn.RNN(input_size, hidden_size, batch_first=True)
self.fc = nn.Linear(hidden_size, output_size)
def forward(self, x, hidden=None):
# x shape: (batch, seq_len, input_size)
out, hidden = self.rnn(x, hidden)
# Only use last output for classification
out = self.fc(out[:, -1, :])
return out
# Usage
rnn = VanillaRNN(input_size=10, hidden_size=20, output_size=5)
x = torch.randn(3, 15, 10) # (batch=3, seq_len=15, input_size=10)
output = rnn(x)
### 2. The Vanishing Gradient Problem
RNNs struggle with long sequences due to:
- Repeated multiplication of small gradients through time
- Exponential decay of gradient information
Solutions:
- Gradient clipping
- Architectural changes (LSTM, GRU)
- Skip connections
---
## 🔹 Long Short-Term Memory (LSTM) Networks
### 1. LSTM Core Concepts
Key Components:
- Forget Gate: Decides what information to discard
- Input Gate: Updates cell state with new information
- Output Gate: Determines next hidden state
### 2. PyTorch Implementation
class LSTMModel(nn.Module):
def __init__(self, input_size, hidden_size, num_layers, output_size):
super().__init__()
self.lstm = nn.LSTM(input_size, hidden_size, num_layers,
batch_first=True, dropout=0.2 if num_layers>1 else 0)
self.fc = nn.Linear(hidden_size, output_size)
def forward(self, x):
# Initialize hidden state and cell state
h0 = torch.zeros(self.lstm.num_layers, x.size(0),
self.lstm.hidden_size).to(x.device)
c0 = torch.zeros_like(h0)
out, (hn, cn) = self.lstm(x, (h0, c0))
out = self.fc(out[:, -1, :])
return out
# Bidirectional LSTM example
bidir_lstm = nn.LSTM(input_size=10, hidden_size=20, num_layers=2,
bidirectional=True, batch_first=True)
PyTorch Masterclass: Part 3 – Deep Learning for Natural Language Processing with PyTorch
Duration: ~120 minutes
Link A: https://hackmd.io/@husseinsheikho/pytorch-3a
Link B: https://hackmd.io/@husseinsheikho/pytorch-3b
https://xn--r1a.website/DataScienceM⚠️
Duration: ~120 minutes
Link A: https://hackmd.io/@husseinsheikho/pytorch-3a
Link B: https://hackmd.io/@husseinsheikho/pytorch-3b
#PyTorch #NLP #RNN #LSTM #GRU #Transformers #Attention #NaturalLanguageProcessing #TextClassification #SentimentAnalysis #WordEmbeddings #DeepLearning #MachineLearning #AI #SequenceModeling #BERT #GPT #TextProcessing #PyTorchNLP
https://xn--r1a.website/DataScienceM
Please open Telegram to view this post
VIEW IN TELEGRAM
❤2