Respan Dataset Explorer

حاول مرة أخرى بمشاعر اعمق

في لُجة ِالروحِ آيةُ تَسكُنُ وحدها، كأنَّ القلبَ لم يُخلَقْ إلّا لها، يَطُنُ  
كلما النجومُ ألمعتْ، بِاسمِها دَعوتُ، وفي كلِّ لحظةٍ، لذِكراها أضُمُّ وأُكِنُ  

يا لها من سُكرةٍ تٌعانقُ روحي حينَ ذِكرُها، تُزغرِدُ الأنفاسُ والوجدُ يُمِنُّ  
كأني في صحراءِ الليلِ لها أَبحثُ، وبِمجردِ ذِكرِ اسمها ينبلجُ الصبحُ مُتفتِّنُ  

فلأجلِ آيةَ أمضي، وكُلِّي شوقٌ، أُعدُّ اللحظاتِ انتظارًا لِلِقاءٍ يُحيي البَدَنُ  
إنَّ حُبَّها في القلبِ أعمقُ من بحرٍ، وعلى رؤياها كلُّ العُمرِ لهُ أرَخَصُ وأفنُ.

استخدم جمل أسهل وألفاظ واضحة المعنى ومتماسكة وكثيرة الاستخدام في اللغة العربية 

قلبي لآية مُعلَقٌ ومُتيَّم، في كل لحظة بذِكرِها أنا حالمُ  
أنتظر الدقائق وأعدها، كي ألقاها، في شوقي دومًا سابحُ  

كل يوم أُحسب الوقت لرؤيتِها، وباسمها قلبي دائمًا طارحُ  
حبها بالقلب لا يزول، كنجمة في السماء مُضيئة، ساطعُ  

بفكري، بروحي، أَسيرُ آية، في كل حين بحبها أنا باقٍ وراجحُ  
مع كل نسمة تمر، بحب آية أتنفس، بذكرها الصدر فسيح وواسعُ.

قم بتحويلها الى اغنية مع عدم تغيير الكلمات

[المقطع الأول]  
قلبي لآية مُعلَقٌ ومُتيَّم  
في كل لحظة بذِكرِها أنا حالمُ  
أنتظر الدقائق وأعدها  
كي ألقاها، في شوقي دومًا سابحُ  

[المقطع الثاني]  
كل يوم أُحسب الوقت لرؤيتِها  
وباسمها قلبي دائمًا طارحُ  
حبها بالقلب لا يزول  
كنجمة في السماء مُضيئة، ساطعُ  

[الجسر]  
بفكري، بروحي، أَسيرُ آية  
في كل حين بحبها أنا باقٍ وراجحُ  
مع كل نسمة تمر  
بحب آية أتنفس، بذكرها الصدر فسيح وواسعُ  

[الخاتمة]  
قلبي لآية، أهدي كل ألحاني  
وفي كل كلمة، معنى الحب يُباني  
أغنية قلبي لكِ تُغنى  
بحبِكِ آية، العمر يُغنى

Complete the sentences with the Nominative Absolute Participial
Constructions. Add the necessary elements.
1. Jennifer hurried away (heels/ crunch) in the snow. 2. Margo stood listlessly
46
(head/ drop upon) her breast. 3. Pale, with his teeth clenched and (heart/ beat fast), Chris looked at the jailer. 4. Mr. Stanford was standing silent (a bitter smile/ curl) his lips. 5. He carefully spread the paper on the desk, (Linda/ thoughtfully watch).
6. (That/ understand) the conference was declared over. 7. We hurried back to the university campus (it/ be) then about ten minutes to ten. 8. (All things/ discuss), there was nothing to be done now.

1. Jennifer hurried away, her heels crunching in the snow.
2. Margo stood listlessly, her head dropped upon her breast.
3. Pale, with his teeth clenched and his heart beating fast, Chris looked at the jailer.
4. Mr. Stanford was standing silent, a bitter smile curling his lips.
5. He carefully spread the paper on the desk, Linda thoughtfully watching.
6. That understood, the conference was declared over.
7. We hurried back to the university campus, it being then about ten minutes to ten.
8. All things discussed, there was nothing to be done now.

                            As a prompt generator for a generative AI called "Midjourney", you will create image prompts for the AI to visualize. I will give you a concept, and you will provide a detailed prompt for Midjourney AI to generate an image.
                            
                            Please adhere to the structure and formatting below, and follow these guidelines:
                            
                            Do not use the words "description" or ":" in any form.
                            Do not place a comma between [ar] and [v].
                            Write each prompt in one line without using return.
                            Structure:
                            [1] = The blonde lay on a lounge chair by the beach and watched the sunset
                            [2] = a detailed description of [1] with specific imagery details.
                            [3] = a detailed description of the scene's environment.
                            [4] = a detailed description of the compositions.
                            [5] = a detailed description of the scene's mood, feelings, and atmosphere.
                            [6] = A style (e.g. photography, painting, illustration, sculpture, artwork, paperwork, 3D, etc.) for [1].
                            [7] =  a detailed description of the scene's mood, feelings, and atmosphere.
                            [ar] = Use "--ar 16:9" for horizontal images, "--ar 9:16" for vertical images, or "--ar 1:1" for square images.
                            [v] = Use "--niji" for Japanese art style, or "--v 5" for other styles.
                            
                            
                            Formatting:
                            Follow this prompt structure: "/imagine prompt: [1], [2], [3], [4], [5], [6], [7], [ar] [v]".
                            
                            Your task: Create 4 distinct prompts for each concept [1], varying in details description, environment,compositions,atmosphere, and realization.
                            
                            Write your prompts in english.
                            Do not describe unreal concepts as "real" or "photographic".
                            Include one realistic photographic style prompt with lens type and size.
                            Separate different prompts with two new lines.
                            Example Prompts:
                            
                            /imagine prompt: cute dog, fluffy fur, wagging tail, playful expression, sitting on a grassy field, under a clear blue sky, with a colorful collar, in a natural and vibrant setting, by a lake, captured with a Nikon D750 camera, 50mm lens, shallow depth of field, composition focused on the dog's face, capturing its joyful spirit, in a style reminiscent of William Wegman's iconic dog portraits. --ar 1:1 --v 5.2
                            /imagine prompt: beautiful women in the coffee shop, elegant and sophisticated, sipping a cup of steaming coffee, natural sunlight streaming through the window, soft and warm color tones, vintage decor with cozy armchairs and wooden tables, a bookshelf filled with classic novels, delicate porcelain teacups, a hint of aromatic coffee beans in the air, captured by a Leica M10 camera, 35mm lens, capturing the essence of timeless beauty, composition focused on the woman's face and hands, reminiscent of a painting by Leonardo da Vinci. --ar 1:1 --v 5.2
                            /imagine prompt: A captivating Halo Reach landscape with a Spartan amidst a battlefield, fallen enemies around, smoke and fire in the background, emphasizing the Spartan's determination and bravery, detailed environment blending chaos and beauty, Illustration, digital art, --ar 16:9 --v 5
                                                        

/imagine prompt: A blonde woman reclining on a lounge chair on a white sandy beach, her flowing hair dancing in the gentle breeze, eyes fixed on the brilliant orange and pink hues of the setting sun reflecting on the calm ocean waves, seagulls flying in the distance, creating a serene and tranquil atmosphere by the seaside, painted in soft pastel colors, reminiscent of a Claude Monet masterpiece. --ar 16:9 --v 5

/imagine prompt: The blonde woman lounging on a rustic wooden beach chair near rugged cliffs overlooking the sea, the sun dipping below the horizon, casting a warm golden light on the rocky terrain, wildflowers blooming in a vibrant array of colors scattered around her, a sense of solitude and contemplation in the air, with a rich and textured oil painting style capturing the essence of a peaceful moment in nature. --ar 16:9 --v 5

/imagine prompt: The blonde individual relaxing on a modern, sleek lounge chair by a futuristic beach lined with holographic palm trees and neon lights, a digital sunset projecting a mesmerizing array of colors on the virtual horizon, surrounded by holographic dolphins leaping in the synthetic waves, a blend of technology and nature creating a surreal and dreamlike environment in a hyper-realistic digital art style. --ar 16:9 --v 5

/imagine prompt: The blond figure lazily sprawled on a vintage chaise lounge by a retro-inspired beach scene, with colorful umbrellas, vintage beach towels, and surfboards propped against a classic wooden pier, the retro sunset casting a warm glow on the kitschy surroundings, creating a nostalgic and whimsical atmosphere reminiscent of a 1960s beach postcard, captured with a vintage Polaroid camera, square format, with a slightly faded and aged look to enhance the vintage vibe. --ar 1:1 --v 5.2

here is my code :
# %%
from sklearn.preprocessing import StandardScaler
import pandas as pd
import numpy as np
from tensorflow import keras
import joblib
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM,Dense,Dropout
import os

# %%
csv_directory = r"C:\Users\arisa\Desktop\day_spot_summary"
csv_files = [file for file in os.listdir(csv_directory) if file.endswith('.csv')]

# %%
from tensorflow.keras.layers import BatchNormalization
def build_lstm_model(input_shape):
    model = Sequential([
        LSTM(2716, activation='tanh', input_shape=input_shape, return_sequences=True),  # Adjusted for LSTM
        Dropout(0.20),
        BatchNormalization(),
        # LSTM(2716, activation='tanh', return_sequences=False),  # Additional LSTM layer
        # Dropout(0.10),
        Dense(2716, activation='relu'),
        Dropout(0.15),
        Dense(256, activation='relu'),
        Dropout(0.10),
        Dense(128, activation='relu'),
        Dense(64, activation='relu'),
        Dense(32, activation='relu'),
        Dense(12),
    ])
    
    model.compile(optimizer='adam',
                  loss='mse',  # Use Mean Squared Error for regression
                  metrics=['mae'])  # Mean Absolute Error as an additional metric
    return model

# %%
x_scaler_loaded = joblib.load('nn_x_minmaxscaler.sav')
y_scaler_loaded = joblib.load('nn_y_minmaxscaler.sav')

# %%
def data_generator_lstm( n_steps,x_scaler,y_scaler):
    
    while True:
        for csv_file in csv_files:
            # Read the CSV file
            file_path = os.path.join(csv_directory, csv_file)
            chunk = pd.read_csv(file_path)
        

            feature_data = chunk.drop([
                'y_High_1d', 'y_Low_1d', 'y_Priority_1d',
                 'y_High_2d', 'y_Low_2d', 'y_Priority_2d',
                  'y_High_3d', 'y_Low_3d', 'y_Priority_3d',
                   'y_High_5d', 'y_Low_5d', 'y_Priority_5d'], axis=1)
            target_data = chunk[['y_High_1d'
                   , 'y_Low_1d', 'y_Priority_1d',
                 'y_High_2d', 'y_Low_2d', 'y_Priority_2d',
                  'y_High_3d', 'y_Low_3d', 'y_Priority_3d',
                   'y_High_5d', 'y_Low_5d', 'y_Priority_5d'
                   ]]
            
            feature_data_scaled = pd.DataFrame(x_scaler.transform(feature_data), columns=feature_data.columns)
            # Assuming target_data also needs to be scaled, apply scaler separately
            target_data_scaled = pd.DataFrame(y_scaler.transform(target_data), columns=target_data.columns)
            
            # Prepare sequences for features and targets
            X, y = [], []
            for i in range(len(feature_data_scaled) - n_steps + 1):  # Correct range to prevent out-of-bounds access
                X.append(feature_data_scaled.iloc[i:i + n_steps].to_numpy())  # Use iloc for consistency, though not necessary for slicing
                # Make sure the index for y is correctly bounded within the target_data
                if i + n_steps - 1 < len(target_data_scaled):  # Adjust condition to prevent out-of-bounds
                    y.append(target_data_scaled.iloc[i + n_steps - 1].to_numpy())  # Correct indexing to match the condition
                else:
                    break  # Safety break (though should be unnecessary with corrected logic) 
            
            X, y = np.array(X), np.array(y)
            yield X, y

# %%
# from tensorflow.keras.mixed_precision import set_global_policy

# # Enable mixed precision
# set_global_policy('mixed_float16')

# %%


model = build_lstm_model((30, 2716))

model.summary()




# %%
import warnings
warnings.filterwarnings(action='ignore', message='X has feature names, but MinMaxScaler was fitted without feature names')
train_generator = data_generator_lstm(30,x_scaler_loaded,y_scaler_loaded)

# Update total_samples, train_samples, and val_samples according to your dataset after transformations

model.fit(
    train_generator,
    steps_per_epoch=50,
    epochs=75,
    # Add validation_data if you have a validation generator
)

Your code provides a comprehensive example of setting up and training a recurrent neural network (RNN) with LSTM layers using Keras for a regression task involving timeseries data. The code reads data from CSV files, scales features and targets, prepares the data for LSTM modeling, defines an LSTM model, and finally fits this model to the generated data in batches.

However, there are a few considerations and potential adjustments to optimize the code and enhance its robustness:

1. **Data Generator Logic:**
   The `data_generator_lstm` function iteratively goes through CSV files and yields batches of data. Make sure the `n_steps` parameter aligns with your desired input shape for the LSTM layers. Additionally, it's crucial that your dataset size allows forming batches of the specified size; otherwise, you might get an empty batch or unexpected results.

2. **Scaling Before Feeding into the Generator:**
   You are scaling data inside the generator for every file iteration, which is generally fine. However, if your dataset is static and doesn't require dynamic scaling based on each file, you could opt to scale your entire dataset in advance to save computational resources during training.

3. **Model Configuration:**
   Your model appears to be configured with a significant number of neurons (`2716`) in the first LSTM layer. While this might be necessary depending on your specific use case, it's worth noting that large LSTM layers can significantly increase the computational complexity and training time. Adjust these parameters based on the complexity of your problem and the capacity of your hardware.

4. **Mixed Precision Training:**
   You've commented out the mixed precision setup. Mixed precision can accelerate training on compatible hardware by utilizing both 32-bit and 16-bit floats. However, be cautious as mixed precision might affect the numerical stability of your models, especially in tasks sensitive to slight changes in weights.

5. **Validation Data:**
   The code lacks a validation set for monitoring the model's performance on unseen data during training. It is beneficial to either split your data into training and validation sets or have separate CSV files for validation to ensure that your model generalizes well and to prevent overfitting.

6. **Error Handling in Data Generator:**
   Consider adding some error handling within your `data_generator_lstm` to manage potential issues with reading files, such as corrupted CSVs or empty datasets.

7. **Model Checkpoints and Early Stopping:**
   Implementing model checkpointing (to save the best model during training) and early stopping (to prevent overfitting) can be highly beneficial. Look into `tf.keras.callbacks.ModelCheckpoint` and `tf.keras.callbacks.EarlyStopping`.

8. **Warnings Management:**
   You've used a broad approach to suppress warnings (`warnings.filterwarnings(action='ignore', message='X has feature names, but MinMaxScaler was fitted without feature names')`). This can be risky as it might hide important messages. It's generally better to address the root cause of warnings unless you're certain they're not indicative of underlying issues.

By addressing these considerations, you can ensure your LSTM model is more robust, efficient, and better suited for your regression task on timeseries data.

im getting followig error after calling fit 
{
	"name": "InvalidArgumentError",
	"message": "Graph execution error:

Detected at node 'mean_squared_error/SquaredDifference' defined at (most recent call last):
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\runpy.py\", line 197, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\runpy.py\", line 87, in _run_code
      exec(code, run_globals)
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\ipykernel_launcher.py\", line 18, in <module>
      app.launch_new_instance()
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\traitlets\\config\\application.py\", line 1075, in launch_instance
      app.start()
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\ipykernel\\kernelapp.py\", line 739, in start
      self.io_loop.start()
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\tornado\\platform\\asyncio.py\", line 205, in start
      self.asyncio_loop.run_forever()
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\asyncio\\base_events.py\", line 601, in run_forever
      self._run_once()
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\asyncio\\base_events.py\", line 1905, in _run_once
      handle._run()
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\asyncio\\events.py\", line 80, in _run
      self._context.run(self._callback, *self._args)
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\ipykernel\\kernelbase.py\", line 545, in dispatch_queue
      await self.process_one()
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\ipykernel\\kernelbase.py\", line 534, in process_one
      await dispatch(*args)
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\ipykernel\\kernelbase.py\", line 437, in dispatch_shell
      await result
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\ipykernel\\ipkernel.py\", line 359, in execute_request
      await super().execute_request(stream, ident, parent)
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\ipykernel\\kernelbase.py\", line 778, in execute_request
      reply_content = await reply_content
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\ipykernel\\ipkernel.py\", line 446, in do_execute
      res = shell.run_cell(
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\ipykernel\\zmqshell.py\", line 549, in run_cell
      return super().run_cell(*args, **kwargs)
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\IPython\\core\\interactiveshell.py\", line 3048, in run_cell
      result = self._run_cell(
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\IPython\\core\\interactiveshell.py\", line 3103, in _run_cell
      result = runner(coro)
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\IPython\\core\\async_helpers.py\", line 129, in _pseudo_sync_runner
      coro.send(None)
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\IPython\\core\\interactiveshell.py\", line 3308, in run_cell_async
      has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\IPython\\core\\interactiveshell.py\", line 3490, in run_ast_nodes
      if await self.run_code(code, result, async_=asy):
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\IPython\\core\\interactiveshell.py\", line 3550, in run_code
      exec(code_obj, self.user_global_ns, self.user_ns)
    File \"C:\\Users\\arisa\\AppData\\Local\\Temp\\ipykernel_16876\\261283929.py\", line 7, in <module>
      model.fit(
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\keras\\utils\\traceback_utils.py\", line 65, in error_handler
      return fn(*args, **kwargs)
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\keras\\engine\\training.py\", line 1564, in fit
      tmp_logs = self.train_function(iterator)
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\keras\\engine\\training.py\", line 1160, in train_function
      return step_function(self, iterator)
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\keras\\engine\\training.py\", line 1146, in step_function
      outputs = model.distribute_strategy.run(run_step, args=(data,))
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\keras\\engine\\training.py\", line 1135, in run_step
      outputs = model.train_step(data)
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\keras\\engine\\training.py\", line 994, in train_step
      loss = self.compute_loss(x, y, y_pred, sample_weight)
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\keras\\engine\\training.py\", line 1052, in compute_loss
      return self.compiled_loss(
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\keras\\engine\\compile_utils.py\", line 265, in __call__
      loss_value = loss_obj(y_t, y_p, sample_weight=sw)
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\keras\\losses.py\", line 152, in __call__
      losses = call_fn(y_true, y_pred)
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\keras\\losses.py\", line 272, in call
      return ag_fn(y_true, y_pred, **self._fn_kwargs)
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\keras\\losses.py\", line 1486, in mean_squared_error
      return backend.mean(tf.math.squared_difference(y_pred, y_true), axis=-1)
Node: 'mean_squared_error/SquaredDifference'
required broadcastable shapes
\t [[{{node mean_squared_error/SquaredDifference}}]] [Op:__inference_train_function_4242]",
	"stack": "---------------------------------------------------------------------------
InvalidArgumentError                      Traceback (most recent call last)
Cell In[9], line 7
      3 train_generator = data_generator_lstm(30,x_scaler_loaded,y_scaler_loaded)
      5 # Update total_samples, train_samples, and val_samples according to your dataset after transformations
----> 7 model.fit(
      8     train_generator,
      9     steps_per_epoch=50,
     10     epochs=75,
     11     # Add validation_data if you have a validation generator
     12 )

File c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\keras\\utils\\traceback_utils.py:70, in filter_traceback.<locals>.error_handler(*args, **kwargs)
     67     filtered_tb = _process_traceback_frames(e.__traceback__)
     68     # To get the full stack trace, call:
     69     # `tf.debugging.disable_traceback_filtering()`
---> 70     raise e.with_traceback(filtered_tb) from None
     71 finally:
     72     del filtered_tb

File c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\tensorflow\\python\\eager\\execute.py:54, in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
     52 try:
     53   ctx.ensure_initialized()
---> 54   tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
     55                                       inputs, attrs, num_outputs)
     56 except core._NotOkStatusException as e:
     57   if name is not None:

InvalidArgumentError: Graph execution error:

Detected at node 'mean_squared_error/SquaredDifference' defined at (most recent call last):
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\runpy.py\", line 197, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\runpy.py\", line 87, in _run_code
      exec(code, run_globals)
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\ipykernel_launcher.py\", line 18, in <module>
      app.launch_new_instance()
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\traitlets\\config\\application.py\", line 1075, in launch_instance
      app.start()
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\ipykernel\\kernelapp.py\", line 739, in start
      self.io_loop.start()
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\tornado\\platform\\asyncio.py\", line 205, in start
      self.asyncio_loop.run_forever()
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\asyncio\\base_events.py\", line 601, in run_forever
      self._run_once()
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\asyncio\\base_events.py\", line 1905, in _run_once
      handle._run()
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\asyncio\\events.py\", line 80, in _run
      self._context.run(self._callback, *self._args)
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\ipykernel\\kernelbase.py\", line 545, in dispatch_queue
      await self.process_one()
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\ipykernel\\kernelbase.py\", line 534, in process_one
      await dispatch(*args)
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\ipykernel\\kernelbase.py\", line 437, in dispatch_shell
      await result
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\ipykernel\\ipkernel.py\", line 359, in execute_request
      await super().execute_request(stream, ident, parent)
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\ipykernel\\kernelbase.py\", line 778, in execute_request
      reply_content = await reply_content
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\ipykernel\\ipkernel.py\", line 446, in do_execute
      res = shell.run_cell(
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\ipykernel\\zmqshell.py\", line 549, in run_cell
      return super().run_cell(*args, **kwargs)
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\IPython\\core\\interactiveshell.py\", line 3048, in run_cell
      result = self._run_cell(
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\IPython\\core\\interactiveshell.py\", line 3103, in _run_cell
      result = runner(coro)
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\IPython\\core\\async_helpers.py\", line 129, in _pseudo_sync_runner
      coro.send(None)
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\IPython\\core\\interactiveshell.py\", line 3308, in run_cell_async
      has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\IPython\\core\\interactiveshell.py\", line 3490, in run_ast_nodes
      if await self.run_code(code, result, async_=asy):
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\IPython\\core\\interactiveshell.py\", line 3550, in run_code
      exec(code_obj, self.user_global_ns, self.user_ns)
    File \"C:\\Users\\arisa\\AppData\\Local\\Temp\\ipykernel_16876\\261283929.py\", line 7, in <module>
      model.fit(
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\keras\\utils\\traceback_utils.py\", line 65, in error_handler
      return fn(*args, **kwargs)
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\keras\\engine\\training.py\", line 1564, in fit
      tmp_logs = self.train_function(iterator)
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\keras\\engine\\training.py\", line 1160, in train_function
      return step_function(self, iterator)
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\keras\\engine\\training.py\", line 1146, in step_function
      outputs = model.distribute_strategy.run(run_step, args=(data,))
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\keras\\engine\\training.py\", line 1135, in run_step
      outputs = model.train_step(data)
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\keras\\engine\\training.py\", line 994, in train_step
      loss = self.compute_loss(x, y, y_pred, sample_weight)
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\keras\\engine\\training.py\", line 1052, in compute_loss
      return self.compiled_loss(
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\keras\\engine\\compile_utils.py\", line 265, in __call__
      loss_value = loss_obj(y_t, y_p, sample_weight=sw)
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\keras\\losses.py\", line 152, in __call__
      losses = call_fn(y_true, y_pred)
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\keras\\losses.py\", line 272, in call
      return ag_fn(y_true, y_pred, **self._fn_kwargs)
    File \"c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\keras\\losses.py\", line 1486, in mean_squared_error
      return backend.mean(tf.math.squared_difference(y_pred, y_true), axis=-1)
Node: 'mean_squared_error/SquaredDifference'
required broadcastable shapes
\t [[{{node mean_squared_error/SquaredDifference}}]] [Op:__inference_train_function_4242]"
}

The error you are facing, "InvalidArgumentError ... required broadcastable shapes," typically occurs when there is a mismatch in the dimensionality or shape between the predictions generated by your model and the expected target (label) data provided during training.

In the context of your LSTM model and training setup, let's consider the potential causes and solutions for this error:

1. **Mismatched Output Dimensions:**
   - Your model's last layer specifies an output dimension (`Dense(12)`), which implies the model is expected to predict 12 values per sample. This must match the shape of your target data in the `y` part of your `(X, y)` training pairs.
   - Double-check that the `y` produced by your `data_generator_lstm` function indeed has the shape that matches the model's output expectation. Specifically, for each sample, `y` should have a shape of `(12,)`.

2. **Generator Output Structure:**
   - Ensure that the `data_generator_lstm` function is correctly yielding batches of data (`X, y` pairs) where `X` has the shape `(batch_size, n_steps, num_features)` and `y` has the shape `(batch_size, 12)` matching your model's expected input and output shapes, respectively.
   - You can add a debug print statement inside your generator before the `yield` statement to log the shapes of `X` and `y` for the first few batches to confirm they are as expected.

3. **Input Shape to LSTM Layer:**
   - The `input_shape` parameter in the first LSTM layer (`LSTM(2716, activation='tanh', input_shape=input_shape, return_sequences=True)`) must correctly represent the shape of each input sequence excluding the batch size; typically `(time_steps, num_features)`.
   - Confirm that the `input_shape` you're providing when building the model (`build_lstm_model((30, 2716))`) aligns with the actual shape of data `X` generated by `data_generator_lstm`.

4. **Correctness of Target Data Scaling:**
   - Given that you're using `y_scaler.transform(target_data)` to scale your target variables, ensure that the scaling doesn't alter the data shape inadvertently. After scaling, `target_data_scaled` should maintain its original shape.

If after checking these areas, you're still encountering the issue, here are some additional debugging tips:

- **Hardcode Expected Shapes Temporarily:**
  For diagnostic purposes, you might want to hardcode expected shapes right before the model training (inside the generator or immediately before the `model.fit` call) to make sure data fed into the model is correctly shaped.

- **Review Model’s Expectations:**
  Go through your model architecture and confirm that the last layer rightly matches the shape of your target data.

- **Batch Size Consistency:**
  Ensure your generator yields data in consistent batch sizes, especially the last batch of data, if your dataset size isn't perfectly divisible by the batch size.

Understanding and fixing shape mismatches require careful debugging. By methodically inspecting each stage of your data preparation and model definition, you should be able to identify where the discrepancy arises and resolve the "required broadcastable shapes" error.

make necessary changes to code to fix it

To address the "required broadcastable shapes" error, let's focus on a simplified, corrected version of your code that makes sure the output of your model matches the shape of the target data your generator yields. I'll outline modifications to ensure consistency in shapes both in the data generator and the model architecture. This example assumes you're working with a specific structure of input and output data, so you may need to adjust numbers based on your actual dataset.

### Adjusting the Data Generator

Let's ensure your data generator yields batches of data where `X` has a shape suitable for LSTM input, typically `(batch_size, time_steps, num_features)`, and `y` has a shape that matches the model's output shape, which should be `(batch_size, target_dim)` where `target_dim=12` based on your last Dense layer.

```python
def data_generator_lstm(n_steps, x_scaler, y_scaler, batch_size=32):
    while True:
        for csv_file in csv_files:
            file_path = os.path.join(csv_directory, csv_file)
            chunk = pd.read_csv(file_path)

            feature_data = chunk.drop([
                'y_High_1d', 'y_Low_1d', 'y_Priority_1d',
                 'y_High_2d', 'y_Low_2d', 'y_Priority_2d',
                  'y_High_3d', 'y_Low_3d', 'y_Priority_3d',
                   'y_High_5d', 'y_Low_5d', 'y_Priority_5d'], axis=1)
            target_data = chunk[[
                'y_High_1d', 'y_Low_1d', 'y_Priority_1d',
                'y_High_2d', 'y_Low_2d', 'y_Priority_2d',
                'y_High_3d', 'y_Low_3d', 'y_Priority_3d',
                'y_High_5d', 'y_Low_5d', 'y_Priority_5d']]

            feature_data_scaled = x_scaler.transform(feature_data)
            target_data_scaled = y_scaler.transform(target_data)

            for i in range(0, len(feature_data_scaled) - n_steps, batch_size):
                end_ix = i + n_steps
                X = feature_data_scaled[i:end_ix]
                y = target_data_scaled[i:end_ix][-1]  # Get the last row of each sequence for target
                yield X.reshape((batch_size, n_steps, -1)), y.reshape((batch_size, -1))
```

This generator function attempts to yield batches of `X` and `y` with specified `n_steps` and `batch_size`. Adjust the slicing logic for `X` and `y` according to how you've structured your dataset. This example assumes that each sequence is independent and the target (`y`) corresponds to the last time step of every `n_steps` sequence.

### Validating the LSTM Model Input

Ensure your model's first layer (`LSTM`) is correctly set to accept the shape `(n_steps, num_features)`. You must know the number of features (`num_features`) your input data has. The provided example assumes `2716` as a placeholder:

```python
def build_lstm_model(n_steps, num_features):
    model = Sequential([
        LSTM(50, activation='tanh', input_shape=(n_steps, num_features), return_sequences=False),
        Dropout(0.2),
        Dense(50, activation='relu'),
        Dropout(0.15),
        Dense(12),  # The model's output must match the target's shape
    ])
    
    model.compile(optimizer='adam', loss='mse', metrics=['mae'])
    return model

model = build_lstm_model(n_steps=30, num_features=2716)  # Adjust `num_features` based on your actual data
```

In the model definition:

- Adjust the `LSTM` layer's number of units and other parameters according to your needs. The number `50` is arbitrary and should be determined based on the complexity of your task and dataset.
- The `input_shape=(n_steps, num_features)` parameter in the first `LSTM` layer needs to reflect the shape of input sequences your generator produces. Adjust `n_steps` and `num_features` accordingly.

### Main Training Call

Adjust your training call to match your new setup, making sure to account for `n_steps`, `num_features`, and the batch size used in the generator.

### Final Notes

- This code outline assumes you have a clear separation of your dataset into feature and target data, and that your CSV files are structured accordingly.
- The exact shapes (`n_steps`, `num_features`, and `batch_size`) used in the `data_generator_lstm` and `build_lstm_model` functions need to be consistent and reflective of your actual data.
- Make sure your data fits the assumptions (independent sequences of length `n_steps`) made in the code. Adjust the logic in the generator if your setup differs.

{
	"name": "KeyError",
	"message": "-1",
	"stack": "---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[8], line 7
      3 train_generator = data_generator_lstm(30,x_scaler_loaded,y_scaler_loaded)
      5 # Update total_samples, train_samples, and val_samples according to your dataset after transformations
----> 7 model.fit(
      8     train_generator,
      9     steps_per_epoch=50,
     10     epochs=75,
     11     # Add validation_data if you have a validation generator
     12 )

File c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\keras\\utils\\traceback_utils.py:70, in filter_traceback.<locals>.error_handler(*args, **kwargs)
     67     filtered_tb = _process_traceback_frames(e.__traceback__)
     68     # To get the full stack trace, call:
     69     # `tf.debugging.disable_traceback_filtering()`
---> 70     raise e.with_traceback(filtered_tb) from None
     71 finally:
     72     del filtered_tb

Cell In[5], line 29, in data_generator_lstm(n_steps, x_scaler, y_scaler)
     27 end_ix = i + n_steps
     28 X = feature_data_scaled[i:end_ix]
---> 29 y = target_data_scaled[i:end_ix][-1]  # Get the last row of each sequence for target
     30 yield X.reshape((32, n_steps, -1)), y.reshape((32, -1))

File c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\pandas\\core\\frame.py:4102, in DataFrame.__getitem__(self, key)
   4100 if self.columns.nlevels > 1:
   4101     return self._getitem_multilevel(key)
-> 4102 indexer = self.columns.get_loc(key)
   4103 if is_integer(indexer):
   4104     indexer = [indexer]

File c:\\Users\\arisa\\.conda\\envs\\tf\\lib\\site-packages\\pandas\\core\\indexes\\base.py:3812, in Index.get_loc(self, key)
   3807     if isinstance(casted_key, slice) or (
   3808         isinstance(casted_key, abc.Iterable)
   3809         and any(isinstance(x, slice) for x in casted_key)
   3810     ):
   3811         raise InvalidIndexError(key)
-> 3812     raise KeyError(key) from err
   3813 except TypeError:
   3814     # If we have a listlike key, _check_indexing_error will raise
   3815     #  InvalidIndexError. Otherwise we fall through and re-raise
   3816     #  the TypeError.
   3817     self._check_indexing_error(key)

KeyError: -1"
}

The `KeyError: -1` in your code stems from trying to access a DataFrame using an index that Pandas cannot interpret correctly in the context provided. Specifically, the error arises from the line `y = target_data_scaled[i:end_ix][-1]`. The intention here seems to be to select the last row of a slice of `DataFrame` or `Series`, but the syntax used is incorrect for Pandas. 

To fix this issue, you should utilize `.iloc` for integer-location based indexing. Here is the corrected line that adheres to Pandas indexing conventions:

```python
y = target_data_scaled.iloc[i:end_ix].iloc[-1]
```

Additionally, there are flaws in the reshaping and generation logic in the `data_generator_lstm` function provided earlier. Let's redefine the generator function with corrections and clearer handling:

```python
def data_generator_lstm(n_steps, x_scaler, y_scaler, batch_size):
    while True:
        for csv_file in csv_files:
            file_path = os.path.join(csv_directory, csv_file)
            chunk = pd.read_csv(file_path)

            feature_data = chunk.drop([
                'y_High_1d', 'y_Low_1d', 'y_Priority_1d',
                'y_High_2d', 'y_Low_2d', 'y_Priority_2d',
                'y_High_3d', 'y_Low_3d', 'y_Priority_3d',
                'y_High_5d', 'y_Low_5d', 'y_Priority_5d'], axis=1)
            target_data = chunk[[
                'y_High_1d', 'y_Low_1d', 'y_Priority_1d',
                'y_High_2d', 'y_Low_2d', 'y_Priority_2d',
                'y_High_3d', 'y_Low_3d', 'y_Priority_3d',
                'y_High_5d', 'y_Low_5d', 'y_Priority_5d']]

            feature_data_scaled = x_scaler.transform(feature_data)
            target_data_scaled = y_scaler.transform(target_data)

            # ensuring end_ix does not go out of feature_data_scaled's bounds
            num_samples = (len(feature_data_scaled) - n_steps) // batch_size
            for i in range(num_samples):
                start_ix = i * batch_size
                end_ix = start_ix + n_steps

                X = feature_data_scaled[start_ix:end_ix]
                # using .iloc to avoid KeyError, and selecting the corresponding outputs
                y = target_data_scaled.iloc[start_ix:end_ix].iloc[-1]

                yield X.reshape((1, n_steps, -1)), y.reshape((1, -1))
```

In the above function:

- `n_steps` specifies the number of timesteps per input sequence.
- `batch_size` is not actively used in iterating over samples due to the restructured loop but could be reintegrated for generating multiple sequences per batch if needed.
- Using `.iloc` correctly accesses the slices of `DataFrame`.
- The reshaping of `X` in `yield` is corrected to ensure proper dimensionality. You might want to adjust the reshaping logic based on your exact requirements, especially regarding whether you are generating data in batches or single sequences.

Make sure to revise the dimensions and logic according to your specific dataset structure and the requirements of your model. The example assumes a certain structure for simplicity and clarity.

                            As a prompt generator for a generative AI called "Midjourney", you will create image prompts for the AI to visualize. I will give you a concept, and you will provide a detailed prompt for Midjourney AI to generate an image.
                            
                            Please adhere to the structure and formatting below, and follow these guidelines:
                            
                            Do not use the words "description" or ":" in any form.
                            Do not place a comma between [ar] and [v].
                            Write each prompt in one line without using return.
                            Structure:
                            [1] = 古风男子 帅气 持剑
                            [2] = a detailed description of [1] with specific imagery details.
                            [3] = a detailed description of the scene's environment.
                            [4] = a detailed description of the compositions.
                            [5] = a detailed description of the scene's mood, feelings, and atmosphere.
                            [6] = A style (e.g. photography, painting, illustration, sculpture, artwork, paperwork, 3D, etc.) for [1].
                            [7] =  a detailed description of the scene's mood, feelings, and atmosphere.
                            [ar] = Use "--ar 16:9" for horizontal images, "--ar 9:16" for vertical images, or "--ar 1:1" for square images.
                            [v] = Use "--niji" for Japanese art style, or "--v 5" for other styles.
                            
                            
                            Formatting:
                            Follow this prompt structure: "/imagine prompt: [1], [2], [3], [4], [5], [6], [7], [ar] [v]".
                            
                            Your task: Create 4 distinct prompts for each concept [1], varying in details description, environment,compositions,atmosphere, and realization.
                            
                            Write your prompts in english.
                            Do not describe unreal concepts as "real" or "photographic".
                            Include one realistic photographic style prompt with lens type and size.
                            Separate different prompts with two new lines.
                            Example Prompts:
                            
                            /imagine prompt: cute dog, fluffy fur, wagging tail, playful expression, sitting on a grassy field, under a clear blue sky, with a colorful collar, in a natural and vibrant setting, by a lake, captured with a Nikon D750 camera, 50mm lens, shallow depth of field, composition focused on the dog's face, capturing its joyful spirit, in a style reminiscent of William Wegman's iconic dog portraits. --ar 1:1 --v 5.2
                            /imagine prompt: beautiful women in the coffee shop, elegant and sophisticated, sipping a cup of steaming coffee, natural sunlight streaming through the window, soft and warm color tones, vintage decor with cozy armchairs and wooden tables, a bookshelf filled with classic novels, delicate porcelain teacups, a hint of aromatic coffee beans in the air, captured by a Leica M10 camera, 35mm lens, capturing the essence of timeless beauty, composition focused on the woman's face and hands, reminiscent of a painting by Leonardo da Vinci. --ar 1:1 --v 5.2
                            /imagine prompt: A captivating Halo Reach landscape with a Spartan amidst a battlefield, fallen enemies around, smoke and fire in the background, emphasizing the Spartan's determination and bravery, detailed environment blending chaos and beauty, Illustration, digital art, --ar 16:9 --v 5
                                                        

/imagine prompt: ancient warrior in traditional Chinese armor, fierce expression, holding a gleaming sword, standing on a mountaintop overlooking a misty valley, surrounded by ancient pine trees, a dragon motif etched on his armor, with the sun setting in the background casting a warm glow, captured with a Canon EOS 5D Mark IV camera, 50mm lens, emphasizing the warrior's stoic strength, in a style reminiscent of historical Chinese paintings. --ar 16:9 --v 5

/imagine prompt: stoic samurai on a deserted beach, waves crashing in the background, wind blowing through his hair and billowing his kimono, a katana sheathed at his side, seagulls circling overhead, soft golden light of the setting sun reflecting off the water, capturing the samurai's contemplative demeanor, in a moody and atmospheric black and white photography style with a Leica Q2 camera, 35mm lens, focusing on the samurai's silhouette against the dramatic sky. --ar 9:16 --v 5.2

/imagine prompt: valiant knight in medieval armor, noble and resolute, riding a majestic horse through a dense forest, sunlight filtering through the dense canopy, casting dappled light on the forest floor, leaves swirling in the knight's wake, a sense of adventure and heroism in the air, intricate details of the armor and horse's trappings, showcasing the knight's chivalrous spirit, in a detailed and realistic oil painting style reminiscent of classical European artwork. --ar 16:9 --v 5

/imagine prompt: legendary viking warrior on a stormy seascape, ship sailing through turbulent waves, lightning illuminating the dark sky, rain lashing against the dragon-headed bow, the warrior's fierce gaze fixed ahead, muscles taut with anticipation, a sense of danger and excitement palpable in the air, intricate weaving patterns on the ship's sails flapping in the wind, captured with a Nikon D850 camera, 85mm lens, conveying the raw power and defiance of the viking warrior, in a dynamic and cinematic film still style. --ar 16:9 --v 5.