USER
Help i keep getting this error after running ONEAPI_DEVICE_SELECTOR="level_zero:0" ZES_ENABLE_SYSMAN=1 accelerate launch g7.py
File "/home/sdp/.local/lib/python3.11/site-packages/torch/nn/functional.py", line 2264, in embedding
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and xpu:0! (when checking argument for argument index in method wrapper_XPU__index_select)
^CKeyboard interruption in main thread... closing server.
Killing tunnel 127.0.0.1:7860 <> https://2919925b734be25df0.gradio.live
Traceback (most recent call last):
File "/home/sdp/.local/bin/accelerate", line 8, in <module>
sys.exit(main())
^^^^^^
tell me how can i make my script run well across all 8 XPU accelerator intel 1550 gpus on this machine
import gradio as gr
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
# Initialize the tokenizer and model
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-14B-Instruct")
# Load the model from safetensors format
model = AutoModelForCausalLM.from_pretrained(
"/scratch-1/q14", # Update this path to your actual model path
torch_dtype=torch.float16,
device_map="auto", # This will distribute across all available GPUs automatically
trust_remote_code=True
).eval()
# Function to generate text using the model
def generate_text(prompt, max_tokens=50, temperature=0.7, top_p=0.9):
# Prepare the input prompt
messages = [
{"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."},
{"role": "user", "content": prompt}
]
# Tokenize the input prompt
input_text = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True
)
inputs = tokenizer(input_text, return_tensors="pt").to("xpu") # Move input to XPU
# Generate output using model
with torch.no_grad():
outputs = model.generate(
inputs["input_ids"],
max_length=max_tokens + len(inputs["input_ids"][0]),
temperature=temperature,
top_p=top_p,
repetition_penalty=1.05,
pad_token_id=tokenizer.eos_token_id
)
# Decode the generated tokens
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
return generated_text
# Create the Gradio interface
iface = gr.Interface(
fn=generate_text,
inputs=[
gr.Textbox(lines=2, placeholder="Enter your prompt here...", label="Prompt"),
gr.Slider(minimum=1, maximum=512, value=50, label="Max Tokens"),
gr.Slider(minimum=0.1, maximum=1.0, value=0.7, label="Temperature"),
gr.Slider(minimum=0.1, maximum=1.0, value=0.9, label="Top P")
],
outputs=gr.Textbox(label="Generated Text"),
title="Qwen 2.5 14B Instruct Text Generator",
description="Generate text using the Qwen 2.5 14B Instruct model with transformers.",
)
# Launch the interface
iface.launch(share=True)
ASSISTANT
The error you're encountering is due to a device mismatch: some tensors are on the CPU while others are on the XPU (`xpu:0`). To run your script across all 8 Intel XPU accelerators (Intel 1550 GPUs) on your machine, you'll need to ensure that:
1. **All relevant tensors are consistently placed on the XPU devices.**
2. **You're correctly configuring `accelerate` to utilize multiple XPU devices.**
Here's a step-by-step guide to resolve the issue and optimize your script for multi-device execution:
### 1. **Update Environment Variables to Select All XPUs**
Currently, your `ONEAPI_DEVICE_SELECTOR` is set to `"level_zero:0"`, which only selects the first XPU device. To utilize all 8 XPU devices, you should specify each device in the selector. Here's how you can modify it:
```bash
export ONEAPI_DEVICE_SELECTOR="level_zero:0,level_zero:1,level_zero:2,level_zero:3,level_zero:4,level_zero:5,level_zero:6,level_zero:7"
export ZES_ENABLE_SYSMAN=1
```
**Explanation:**
- `ONEAPI_DEVICE_SELECTOR`: This environment variable specifies which XPU devices to use. By listing all devices (`level_zero:0` through `level_zero:7`), you're enabling `accelerate` to recognize and utilize all 8 devices.
- `ZES_ENABLE_SYSMAN=1`: This enables Level Zero system management, which is necessary for monitoring and managing the XPU devices.
You can set these variables directly in your shell before running your script or include them in your launch command.
### 2. **Configure `accelerate` for Multi-Device Setup**
Before proceeding, ensure that `accelerate` is properly configured to handle multiple devices. If you haven't already initialized an `accelerate` configuration, do so now:
```bash
accelerate config
```
During the configuration:
- **Compute Environment**: Choose `Multi-Processing` or the appropriate option that suits your setup.
- **Distributed Type**: Select `Multi-GPU` or `Multi-Accelerator` based on what's available.
- **Number of Processes**: Set this to `8` to match your 8 XPU devices.
- **Processes Per Host**: Typically `1` unless you have a specific multi-host setup.
- **Main Device Port**: Leave as default unless there's a port conflict.
- **Use FP16?**: Yes, since you're using `torch.float16`.
This sets up `accelerate` to manage multiple devices effectively.
### 3. **Modify Your Python Script to Leverage `accelerate`**
To ensure all tensors are placed on the correct devices and to utilize multiple accelerators, integrate `accelerate`'s `Accelerator` class into your script. Here's how you can modify your script:
```python
import gradio as gr
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from accelerate import Accelerator
# Initialize the accelerator
accelerator = Accelerator()
# Initialize the tokenizer and model
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-14B-Instruct")
# Load the model from safetensors format
model = AutoModelForCausalLM.from_pretrained(
"/scratch-1/q14", # Update this path to your actual model path
torch_dtype=torch.float16,
device_map="auto", # This will distribute across all available XPU devices
trust_remote_code=True
).eval()
# Prepare the model with accelerator
model = accelerator.prepare(model)
# Function to generate text using the model
def generate_text(prompt, max_tokens=50, temperature=0.7, top_p=0.9):
# Prepare the input prompt
messages = [
{"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."},
{"role": "user", "content": prompt}
]
# Tokenize the input prompt
input_text = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True
)
# Tokenize and prepare inputs
inputs = tokenizer(
input_text,
return_tensors="pt"
)
# Move inputs to the appropriate device(s)
inputs = {k: v.to(accelerator.device) for k, v in inputs.items()}
# Generate output using model
with torch.no_grad():
outputs = model.generate(
inputs["input_ids"],
max_length=max_tokens + inputs["input_ids"].shape[1],
temperature=temperature,
top_p=top_p,
repetition_penalty=1.05,
pad_token_id=tokenizer.eos_token_id
)
# Decode the generated tokens
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
return generated_text
# Create the Gradio interface
iface = gr.Interface(
fn=generate_text,
inputs=[
gr.Textbox(lines=2, placeholder="Enter your prompt here...", label="Prompt"),
gr.Slider(minimum=1, maximum=512, value=50, label="Max Tokens"),
gr.Slider(minimum=0.1, maximum=1.0, value=0.7, label="Temperature"),
gr.Slider(minimum=0.1, maximum=1.0, value=0.9, label="Top P")
],
outputs=gr.Textbox(label="Generated Text"),
title="Qwen 2.5 14B Instruct Text Generator",
description="Generate text using the Qwen 2.5 14B Instruct model with transformers.",
)
# Launch the interface using accelerator's launch method if necessary
if __name__ == "__main__":
iface.launch(share=True)
```
**Key Modifications:**
1. **Initialize the Accelerator:**
```python
from accelerate import Accelerator
accelerator = Accelerator()
```
This sets up the `Accelerator` object to manage device placement and distributed training.
2. **Prepare the Model with Accelerator:**
```python
model = accelerator.prepare(model)
```
This ensures that the model is correctly distributed across all available XPU devices as per the `accelerate` configuration.
3. **Move Inputs to the Correct Device:**
Instead of manually moving inputs to `"xpu"`, use the accelerator's device:
```python
inputs = {k: v.to(accelerator.device) for k, v in inputs.items()}
```
This ensures that all input tensors are placed on the same device(s) as the model.
4. **Launch with `accelerate`:**
Ensure you're launching the script using `accelerate launch` to leverage the multi-device setup:
```bash
accelerate launch g7.py
```
Since you set the environment variables earlier and configured `accelerate`, this command will handle the multi-device distribution.
### 4. **Run the Script**
With the environment variables set and the script modified, run your script using the `accelerate` launcher:
```bash
ONEAPI_DEVICE_SELECTOR="level_zero:0,level_zero:1,level_zero:2,level_zero:3,level_zero:4,level_zero:5,level_zero:6,level_zero:7" ZES_ENABLE_SYSMAN=1 accelerate launch g7.py
```
**Alternative:** If you've exported the environment variables in your shell, you can simply use:
```bash
accelerate launch g7.py
```
### 5. **Additional Tips**
- **Verify Device Allocation:**
Ensure that all XPUs are being utilized. You can add logging to check where different parts of the model are placed.
- **Monitor Resource Usage:**
Use Intel's profiling tools or other system monitors to verify that all 8 XPUs are actively being used during model inference.
- **Batch Processing:**
For optimal performance across multiple devices, consider implementing batch processing of prompts if applicable.
- **Error Handling:**
If you encounter further errors related to device placement, consider using `accelerator.print(...)` for debugging or consult the `accelerate` documentation for advanced configurations.
By following these steps, you should be able to resolve the device mismatch error and efficiently run your script across all 8 Intel XPU accelerators on your machine.